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ABSTRACT 



Work undertaken to improve the effectiveness of the United 
Kingdom's schools and further education (FE) sectors was identified and 
assessed in a study entailing four data collection methods: literature 
review; questionnaire administered to all FE college principals in England 
and Wales; expert seminar and face-to-face interviews with high-level 
representatives of all major agencies responsible for the schools, higher 
education (HE) , and FE sectors; and focus group and telephone interviews with 
quality managers in a sample of colleges. Although the questionnaire elicited 
a response rate of only 18%, the respondents were representative of the FE 
sector as a whole. Among the main obstacles to improving sector effectiveness 
identified by the principals were the following: lack of finance and funding; 
lack of management systems and training for middle managers; and need for 
lecturing staff to adopt new teaching methods. Confusion over the terms 
"effectiveness" and "improvement" was discovered. It was concluded that 
systematic quantitative and qualitative research based on a theoretical 
framework clarifying its relationship to Further Education Funding Council 
policy and practice is critical to future work to improve the schools, FE, 
and HE sectors. (Appended are the following: research methodology, 
bibliography that contains 125 references, and information about the study's 
authors.) (MN) 
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Foreword 



Since incorporation, one of the main drivers for colleges has been the need 
to improve efficiency. This has been expressed through the very detailed 
requirements of the funding regime: efficiency targets, convergence, and the 
allocation and claw back mechanisms. 

By contrast, the driver to improve effectiveness has been blunt: the direct 
linking of funding to retention and achievement. 

There is now a professional and political consensus that we need to be more 
effective. In particular, we need to improve retention and achievement, both 
rapidly and significantly. 

Against this background, this report is particularly welcome and timely. 
It contains a robust and, indeed, critical review of previous research, 
inspection and developmental activities, and current perceptions of the issue 
within the sector. It confronts difficulties and complexities head on. In 
addition, it summarises very succinctly the huge amount of work that has 
already been done on effectiveness and improvement in the school sector 
and makes this accessible to colleges. 

Above all, I welcome the rigorous analysis of problems and the recommen- 
dations for future research and development. If understanding a problem is 
halfway to resolving it, then this report takes us some way forward. 
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Raising standards is at the forefront of the Government’s agenda for further 
education. At the same time, colleges have been set new and demanding 
recruitment targets to widen participation and encourage lifelong learning. 

In his letter to the FEFC in December, David Blunkett said: 

Raising standards is absolutely essential if the sector is to make the 
contribution to the lifelong learning agenda for which the Comprehensive 
Spending Review settlement provides. Rates of retention and achievement 
must be raised significantly, college by college, course by course and 
year by year. 

The recommendations in this report help us focus on an agenda for research 
that will unite all institutions, managers and teachers in the sector: raising 
student achievement. 

FEDA, with AoC are taking forward many of the recommendations 
in a three-year programme funded by the DfEE to raise quality and 
effectiveness. The Raising quality and achievement programme offers support 
strategies that include: 

• a quality information and advice service 

• quality improvement teams 

• benchmarking and information 

• development projects 

• leadership and governance for achievement 

• best practice 

• research into and evaluation of strategies that work. 

Chris Hughes 
Chief Executive, FEDA 
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Summary 



The aims of this study were to: 

* identify relevant work on improving effectiveness from the 
FE and school sectors, and assess its relevance 

® define the agenda and make recommendations for future work. 

This report outlines the major changes in policy since the Further and 
Higher Education Act (1992). There is a wide range of colleges in the sector 
and a lack of agreement on their purpose and, therefore, no consensus on 
what constitutes an effective or improving college. Methods of measuring 
effectiveness often depend on comparing like with like; this is difficult where 
the purpose, curricula and client groups are extremely diverse within and 
between colleges. 

Principals surveyed for the project see the following as the main obstacles to 
improving effectiveness: 

• the lack of finance and funding which has led to poor pay and 
conditions and low staff morale 

® the lack of management systems and training for middle managers; 
the increase in bureaucracy 

® the need for lecturing staff to adopt new methods of teaching, 
including the use of IT in college and for distance delivery. 
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Definition of terms 



There is confusion about the terms ‘effectiveness’ and ‘improvement’. 
Efficiency is not the same as effectiveness. The former is concerned with the 
relationship between inputs and outputs, whereas the latter is concerned 
with whether the objectives or purposes of the institution have been 
achieved. Improvement implies change over time. An improving institution 
may not be an effective institution, and effective institutions may not be 
efficient or cheap. 

The focus for improving effectiveness should be the teacher, because it is 
relatively easy to relate outcomes to teaching. However, there has to be a 
generally agreed definition of quality, and this can change over time. 

Evidence from research 

There is an extensive body of research relating to the schools sector within 
the specific research paradigm known as ‘school effectiveness’ research 
(SER). Its defining features are that: it is concerned with the outcomes of 
education; institutions should be compared like-with-like; and processes are 
mainly of interest insofar as they are related to outcomes. 

There is increasing interest in research into how schools improve. Schools 
that start to change have usually worked on: 

• the school’s attitude and approach to planning 

• the way the school is run and organised 
® the way the curriculum is organised 

• the ethos or culture of the school 

® aspects of the quality of teaching and learning. 

Few comparable research studies relate to further education. Gray et al. 
(1995) showed that about 11% of 16-19 institutions appear to make a 
difference, and that the kind of institution appears to have some effect on 
students’ examination performance. Conway (1997) describes how GCSE 
scores can be used to predict students’ performance, can improve the work 
of departments that are underperforming, and can be combined with 
valuing each student as an individual. 
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A diffuse body of literature touches on educational effectiveness and improvement 
issues in relation to further education. These studies suggest that self- 
assessment is a key component of quality improvement, and that leadership 
and organisational development are important factors in college effectiveness. 
Developing the culture of a learning organisation depends upon allowing 
staff to grow, and is different from a ‘right first time’ culture. 

The student drop-out rate is a problem. Further work needs to be done on 
developing the curriculum (particularly relating to the issue of ‘parity of 
esteem’ between A-level and GNVQs), retention and value added, since a single 
methodology cannot currently be applied to further education. There is 
renewed interest in teaching and learning as the core component of college 
effectiveness. 

Systems for inspection and improvement 

The ways in which inspection is designed to improve effectiveness in the 
schools and FE sectors are similar, and there appears to be a convergence in 
approach between the FEFC, FEFCW and the Office for Standards in 
Education (OFSTED). There is a variation in emphasis between England 
and Wales, but both approaches are designed to produce a self-critical 
culture that will lead to improvement. 

Of the factors identified by OFSTED as affecting pupil attainment, the most 
important is effective teaching, which has four times the effect of other 
significant factors such as internal and external relationships, quality 
of management and the quality of assessment. Current policy for the FE 
sector places more emphasis on student attainment, matching students to 
appropriate courses, and improving the quality of teaching and learning. 

Improving college effectiveness 

It is essential that future research and development work on improving college 
effectiveness uses a theoretical framework that clarifies its relationship to 
FEFC policy and practice. Frameworks can be categorised, in terms of their 
characteristic assumptions and strategies, according to the extent to which 
they belong to the effectiveness model or the improvement model. Although 
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they contain elements of both models, FEFC procedures for funding and 
inspection currently fall rather more into the effectiveness paradigm than 
the improvement paradigm. 

Funding concerns and low morale are sapping the energy and drive to improve 
effectiveness. This can be addressed by adopting strategies characteristic of 
the improvement paradigm in relation to: the model of change, the culture 
of the organisation and motivation of lecturing staff. The challenge is to do 
this while retaining strategies characteristic of the effectiveness paradigm in 
relation to other areas. 

Research and development work needs to draw on the approaches of both 
paradigms, as they compensate for each other’s weaknesses. This work 
needs to include the key indicators that can be used to judge effectiveness 
and improvement. 

Recommendations 

There is an urgent need for both quantitative and qualitative research, 
carried out systematically with representative samples of colleges, to inform 
the development of policy and practice. 

Research in the effectiveness paradigm carried out with colleges, using 
methods already proven to be robust with the much larger cohort of schools, 
will be of value in establishing trends: 

• Longitudinal studies should be set up to compare ‘like now’ with ‘like 
past’ and ‘like future’. These should involve a large sample of colleges to 
provide a systematic longitudinal study of the sector. 

• There is also a need for studies which establish comparison of like 
with like; establish key indicators; enable value added to be taken into 
consideration; and enable improvement to be monitored. 

This study has identified major gaps in the research so far carried out within 
the improvement paradigm. There is an urgent need for a firm knowledge 
base about the sector, in particular: 

® high-quality case study research to be carried out systematically 
across a representative sample of colleges 
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® studies to monitor outcomes resulting from specific interventions 
® a range of other studies on improvement in infrastructure 
support, in management and organisational structures, and 
in teaching and learning. 

There is a need for an agency to provide leadership for development work in 
the FE sector, distinct from the monitoring role that is part of the FEFC’s 
inspection remit, including: 

• involvement of college staff in research as a form of staff development 
® guidelines for carrying out research, to develop methods for 

supporting improvement in practice 

• establishment of an improvement network of colleges to 
involve all colleges and to enable collaboration with a ‘like’ 
partner and dissemination of good practice between more 
and less effective colleges 

® provision of support and consultancy at the request of colleges, 
to help them improve their effectiveness 

• identification and assistance with the recording of good practice; 
development of strategies for disseminating good practice; and 
establishment of a database of examples of good practice and 
related research. 
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Introduction 



Aims and purposes of the study 

During the period April to June 1998, we carried out a scoping study called 
hnpToving college effectiveness in England and Wales. The study aimed to: 

• identify relevant work on improving effectiveness from 
the FE and school sectors and assess its relevance 

• define the agenda for future work on improving college effectiveness 

• recommend a set of options for future research and 
development activities. 

This is an area of considerable importance because of the economic imperative 
to improve the level of skills and competence in the workforce. International 
comparisons show that the UK lags behind most other developed countries 
in the numbers of skilled technicians and the level of qualifications of blue 
collar workers generally (Otter, 1996). High levels of unemployment and 
low achievement among school leavers have given rise to problems of social 
exclusion; current Government policy places importance upon education 
as a means of combating these problems. The rapidly changing needs of 
society at the end of the twentieth century have led to the new concept of 
lifelong learning as a right of all citizens regardless of age. Further education 
must rise to all of these challenges. 
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As a result of these demands, the last year has seen the publication of influential 
reports that address issues relating to further education’s development. 
Further education provides for all students aged 16 or over who are not in 
schools or HE institutions. In her introduction to Learning Works: widening 
participation in further education (FEFC, 1997a), which sets out the conclusions 
of the committee that she chaired, Helena Kennedy writes: 

Defining further education exhaustively would be God's own challenge 
because it is such a large and fertile section of the education world. Yet, 
despite the formidable role played by further education, it is the least 
understood and celebrated part of the learning tapestry. (Kennedy, 1 997) 

This study sets out to improve understanding of further education’s current 
role and future potential through an investigation of the factors and 
mechanisms that define its effectiveness. 



Policy: context and trends 

The Further and Higher Education Act (1992) brought about a revolution in 
the governance and funding of further education. Colleges were removed 
from the control of LEAs, and responsibility for their funding and provision 
became the province of the newly established FEFC. In addition to general 
and specialist FE colleges, the 1992 Act transferred responsibility for sixth- 
form colleges from LEAs to the FEFC. This group of colleges, traditionally 
closely related to schools, has to an extent retained a different culture from 
the rest of the sector. They are frequently described as being akin to 
grammar schools in ethos and traditions. Certainly for the purposes of this 
study they pose very different issues. 

Since 1992, there has been an increase of about one-third in the number of 
student enrolments in FE colleges, and a 27% reduction in funding for full- 
time equivalent (FTE) students. Other notable changes following the act 
have included new staff contracts and new national data collection and 
monitoring systems established by the FEFC. 



The adoption of the National Targets for Education and Training in 1991, 
followed by the White Paper Competitiveness: Helping Business to Win 
(DTI, 1994) created the need for improvements in the performance of the FE 
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sector, for the first time in relation to measurable targets. The review of these 
targets in 1995, and the report on progress published in 1997 (NACETT, 
1997), continue to pose considerable challenges to both employers and 
colleges. The Conservative Government’s White Paper Learning to Compete 
(DfEE, 1997) was the first ever on 14-19 education and laid the foundation 
for current ambitious policies. 

Since the election of a Labour Government in May 1997, ‘education, 
education and education’ has become the top priority. Two significant 
reports, commissioned by the previous Government, have focused attention 
on the need to extend provision of further education. The report of the committee 
chaired by John Tomlinson, Inclusive Learning (FEFC, 1996c), brought it to 
public attention that the quality of learning opportunities for disabled 
students and those with learning difficulties was poorer than for other students; 
and that some groups of such learners were effectively deprived of the 
opportunity to attend college. In articulating the benefits of inclusive 
learning, the report properly establishes that making provision for students 
with learning difficulties and/or disabilities is an economic imperative as 
well as a moral one. The report identified the need for nationally planned 
and funded staff development, for both managers and teachers, as the most 
urgent of its recommendations. It called for the strengthening of inspection 
arrangements ‘. . . so that they can provide evidence of the match between 
student needs and a college-wide inclusive environment’. Tomlinson places 
the needs of those who had been generally neglected in society firmly on the 
quality and effectiveness agenda of further education. 

Learning Works (FEFC, 1997a), published since the change of Government, 
recommended ‘prioritising widening participation in the post- 16 education 
agenda’ and ‘redistributing public resources towards those with less success 
in earlier learning, moving towards equity of funding in post-16 education’. 
In responding to both the Tomlinson and the Kennedy reports, the FEFC 
has published guidance (FEFC, 1997b) to assist colleges in identifying and 
addressing needs through gathering market intelligence; through the use of 
community profiles to provide a comparator with the current student body; 
and by ensuring the incorporation of needs and market analysis in strategic 
planning processes. 

Both the Tomlinson and the Kennedy reports received an enthusiastic 
welcome from the sector. The Kennedy report, in particular, provided a 
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vision for the sector as a whole and celebrated its achievements. At a time 
when many staff felt that resources had been cut to the bone, Kennedy 
restated values which they hold dear: 

The hallmark of a college’s success is, as it should be, public trust, 
satisfaction of the ‘stakeholders’ and esteem rather than profitability. 
These colleges do not see their students merely as ‘consumers’, or 
learning merely as ‘training’. They see education as being more than 
the acquisition of knowledge and skills. In a system so caught up in 
what is measurable, we can forget that learning is also about problem 
solving, learning to learn, acquiring the capability for intelligent choice 
in exercising personal responsibility. It is a weapon against poverty. 

It is a route to participation and active citizenship. 

The Government’s response to Kennedy was a positive one (DfEE, 1998b): 
the commitment of £100 million additional funding to provide for an 
anticipated extra 80,000 students in further education, . . the great majority 
to be drawn from the educationally disadvantaged population’. Significantly 
for the development of the sector, the Government expressed its belief: 

. . . that excessive emphasis in the past on market competition has 
inhibited collaboration; and that strong partnerships are now needed 
to develop efficient local strategies for learning. 

The 1997 report of the National Advisory Group for Continuing Education 
and Lifelong Learning, chaired by Bob Fryer, Learning for the Twenty-first 
Century, and the subsequent DfEE (1998a) consultation paper The Learning 
Age extend the debate to include the more general need to provide lifelong 
learning opportunities for all adults. Although encompassing a wider range 
of issues than those relating exclusively to the EE sector, these reports have 
immense implications for further education. The most important strategies 
include 500,000 additional people in further and higher education, the New 
Deal for the young and long-term unemployed, the University for Industry 
(Ufl), Individual Learning Accounts and the National Grid for Learning. 
Every one of these will depend to an extent on further education for its 
implementation. While there will be some additional funding, it is generally 
recognised that colleges will be expected to improve their effectiveness to 
ensure that all these policies are delivered. 
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Chapter 5 of the Green Paper Ensuring Standards, Quality and Accountability 
sets out the vision that all learners: 

. . . should be entitled to high quality learning that: delivers what it 
promises; gets them to their goals; and takes them as high up the ladder 
of achievement as they are able to go. 

Regarding further education, the Green Paper says that Government proposes 
to work with the FEFC and FED A to get all colleges to: 

® ‘adopt a rigorous approach to standards, with systematic 
assessment and target setting’ 

® ensure better teaching, to ensure that all new teachers acquire a 
‘recognised teacher training qualification’ 

® ensure both flexibility and continuity by a proper balance 
of full-time and part-time teachers in the workforce 
® ‘promote improvements in college management’. 

Most recently, the report of the House of Commons Select Committee for 
Education and Employment (1998) has supported the development of 
further education along the lines recommended by Kennedy, and has called 
for further funding to enable this. It also endorses Government plans for a 
national qualifications framework and a common modular structure across 
A-levels and GNVQs, with individual certification of modules, to allow 
learners to build up credit towards qualifications that match their needs. 
This is broadly in line with the recommendations of the Bearing report. 
Review of Qualifications for 16-19 Year Olds (1996), and will make it 
easier to monitor the effectiveness of colleges. 

Together, these reports and policy documents present a new agenda for 
further education, in line with the current Government’s vision for education. 
They contain a new emphasis on inclusive education to serve the needs of a 
wide range of stakeholders. The old agenda of competition in the market 
place has been replaced by an agenda of partnership and collaboration, 
coupled with challenging targets to ensure that further education, like other 
sectors of education, ensures the maximum possible achievement for its students. 
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Key points 



• There has been an increase of about one-third in the number of students 
in further education since 1992, without any comparable increase in 
resources. 

• There is a drive to improve performance in the FE sector using systematic 
assessment and target setting; there is a corresponding need for improved 
management techniques and better qualified staff. 

• There is a need to improve the quality of learning opportunities for disabled 
students and those with learning difficulties, and to widen participation 
in further education; there is a corresponding need for staff development 
for managers and lecturers. 

• The Government has made additional funding available for an increase in 
student numbers and further funding is likely to be forthcoming. 

• The need to provide lifelong learning opportunities for adults will be 
met through initiatives such as the New Deal, the Ufl, and Individual 
Learning Accounts, and through changes in the curriculum and qualifications 
framework. 

® There is a need to establish an agenda that stresses partnership and 
collaboration within the FE sector based on challenging targets related to 
the maximisation of student achievement. 
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The state of the colleges 




The FEFC and FEFCW have divided FE sector colleges into six broad types: 

• general FE and tertiary colleges (285 in 1997 - 64% of the total) 

• sixth-form colleges (109 in 1997 - 24.5% of the total) 

• agricultural and horticultural (30 in 1997 - 7% of the total) 

• art and design (7 in 1997- 1.5% of the total) 

• designated colleges (with mainly adult provision) 

(13 in 1997 - 3% of the total) 

• colleges in Wales (26 in 1997). 

Perhaps it is not surprising that, given the wide range of kinds of colleges in 
the sector, there is lack of agreement over their purpose. This results in a lack 
of agreement about what constitutes effectiveness. 

Immediately prior to incorporation, Rick Bearing (1994) identified ‘the 
public service ethic’ as still dominant among most FE staff. Two years later 
Halliday (1996) contrasts the value laden mission of traditional FE colleges 
with ‘. . . the perception that further education is a value-neutral commercial 
response to a presumed market in education’. Policy-makers tend to assume 
the latter. One of our respondents sees further education as ‘a production 
activity’ in which students have the right to acquire the qualification for 
which they enrol. He believes there should be ‘zero tolerance of failure’ and 
has little time for those who perceive student participation to be an end in 




Vol2 No 12 



fed a report 2 0 



19 



itself. In support of his views, he points out that nationally the average pass 
rate for those enrolling on a course in further education is probably as low as 
50%; although pass rates for students who complete a course are generally 
higher, in some colleges they can be as low as 25%. 

The question here is whose values count and whose interests do they serve? 
The argument that students are being ‘robbed’ if a college takes their money 
and fails to deliver a qualification is persuasive. Students may also be ill 
served if they are enrolled on courses with little prospect of leading to 
employment. At present, this happens too often as a result of the pressure on 
colleges to meet financial targets. Vulnerable students who are not from 
educationally aware backgrounds need advocates who will act in their 
interests. Another respondent pointed out that few teachers, managers or 
decision-makers in further education use the sector for significant aspects of 
their own vocational education or that of their offspring. In this sense, FE 
policies could be seen as ‘middle-class schemes for working-class learners.’ 
He wondered if this might have contributed to ‘a situation wherein poor 
retention and achievement have rarely been challenged and mediocrity has 
been widely accepted.’ 

The lack of consensus on purpose among FE colleges is most obvious in 
the contrast between sixth-form colleges. The latter perceives academic 
achievement as the pinnacle of their provision, while general colleges tend to 
look for a broader range of goals, including social cohesion and inclusiveness, 
and a concern with citizenship and Europeanisation, as well as traditional 
academic values. Methods of measuring effectiveness generally rely on 
comparing like with like. This is difficult in a sector with such diversity. 

The divisions lie in the curriculum itself. GNVQs were established to 
provide a competence-based alternative to the ‘gold standard’ of A-levels, 
but it has not been easy to establish their credibility. Despite frequent pleas 
from Government for equal status for vocational qualifications, not all 
universities give them recognition equal to A-levels. Research by Fitz-Gibbon 
(1997) suggests that the four most popular Advanced GNVQs are not entirely 
comparable with A-level because of differences in the philosophy and 
methods of assessment. These differences account in part for the difficulty of 
effectively deploying value added measurements in the mainstream of 
further education (Barnard and Dixon, 1998). 
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Colleges need to serve multiple stakeholders: young people, employers, 
those wishing to return to learning in late youth, middle age or later, local 
councils, TECs and central Government. However, the FEFC report on 
college responsiveness (FEFC, 1996a) suggests that, while colleges are 
broadly successful in responding to student demands and to community 
needs, they appear to give less priority to the needs of employers. Open 
learning facilities were found to be often under-used; few colleges surveyed 
employers’ views; the move toward flexible delivery of modularised courses 
was slow; and marketing activities were, by and large, not properly 
evaluated. Links between FE colleges and employers are extensive in terms 
of college-workplace links at a functional level. However, many colleges 
have surprisingly few links with employers at the level of chief executive or 
managing director. It seems that employers have mainly established these 
links with schools and higher education, partly as a result of Government 
initiatives that have not included further education. 

Since incorporation there have been great changes in the management of FE 
colleges. Colleges are now often perceived by their principals as business 
organisations whose business happens to be education. All colleges have 
mission statements and, following the publication of the Charter for Further 
Education (DFE, 1993), all colleges have their own charters as a condition 
of FEFC funding. Colleges are also involved in quality assurance procedures. 
As early as 1990, a survey (Sallis, 1990) showed that BS5750 had been adopted 
by 39% of respondents, and a further 33% were developing total quality 
management (TQM) approaches. This trend has continued. IS09000 has since 
replaced BS5750. Meanwhile, most colleges are funded for some of their 
work by TECs, which are required by law to ensure compliance with TEC Quality 
Assurance: Supplier Management (Employment Department, 1993). 

There is also widespread take-up of Investors in People. This award is made 
to organisations that can demonstrate: 

• a commitment to develop employees 

• an openly available business plan that incorporates a training policy, 
regular reviews of training and development for all the workforce 

® action to train and develop individuals 

• evaluation of investment in training and development. 
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In spring 1995, a special ‘Quality For All’ edition of the NATFHE Journal 
was devoted specifically to a critical review of what the editorial referred to 
as . the maze of what has become a booming business of assessment and 
measurement’. Many writers have expressed concern regarding the impHcations 
of an uncritical adoption of ‘managerialist’ approaches, labour casualisation, 
inappropriate inspection and audit regimes, funding mechanisms based on 
crude performance indicators, and industrially derived quality ideologies 
(Elliott, 1993; Holmes, 1993; Elliott and Hall, 1994; Reeves, 1995; Avis, 
1996; Elliott, 1996a; Ainley and Bailey, 1997; Randle and Brady, 1997; 
Avis, 1998). 

In our survey of college principals, the areas they raised as major issues in the 
drive to improve college effectiveness were: finance and funding, staffing, 
management and quality assurance, and teaching and learning. What one 
principal called ‘the downward convergence of finance’ was generally 
agreed to have led to ‘the worsening pay and conditions of FE lecturers’, 
resulting in ‘demoralised staff’. The most frequent comments on management 
and quality assurance involved the use and development of management 
information systems (MIS) and the training of middle managers. Comments 
on teaching and learning were exclusively about the need for staff to adopt 
new methods, including the use of IT for both college-based and home delivery. 

The most frequent complaint from college lecturers is about the increase in 
bureaucracy and, specifically, the huge increase in paperwork in recent years. 
According to a senior civil servant with responsibility for further education, 
the volume of paperwork that colleges are producing has increased consid- 
erably. Some colleges that have achieved good inspection results have ‘set up 
amazing systems with volumes and volumes of paper’. There is value in this 
accurate information on what is being done, but clearly it is also taking up 
time that might be spent on other things. 

There appears to have been a corresponding falling away of professional 
development opportunities. Betts (1996), following an analysis of FEFC 
inspection reports and a review of appraisal schemes in seven colleges, 
argues that staff development in further education is not a high priority and 
suffers from short-termism. Ollin (1996) points to the dangers of current 
staff development practices in further education leading to the deskilling of 
lecturers as a consequence of college managers’ failure to learn lessons from 
industry’s approaches to investment in human resource development. 



2 3 



22 



fed a report 



Vol2 No 12 



Key points 

® There is a wide range of colleges in the sector and a lack of agreement on 
their purpose. 

® There can be no agreement, therefore, on what constitutes an effective or 
improving college. 

® Methods of measuring effectiveness often depend on comparing like with 
like; this is difficult where the purpose, curricula and client groups within 
colleges are very diverse. 

» Many principals now see themselves as managers of a business; many 
colleges have adopted quality assurance systems. 

• The major issues that college principals see as obstacles to improving 
effectiveness are: lack of finance and funding, which has led to poor pay 
and conditions for staff; lack of management systems; the lack of training 
for middle managers; increased bureaucracy; and the lack of continuing 
professional development for lecturing staff. 




Definition of terms 



As noted in the previous chapter, there appears to be a lack of agreement 
about the purposes of colleges and what constitutes effectiveness. Before 
going any further, it may therefore be useful to define some of the terms we 
will be using. The following definitions are based on Managing Colleges 
Efficiently (DESAJf'0, 1998): 

® efficiency is based on the relationship of inputs to outputs 
® effectiveness is the extent to which objectives are achieved. 

Birch and Latcham (1985a) explain that: 

... an organisation is effective if it achieves its objectives which are 
appropriate to the needs of society. It is efficient if it achieves these 
objectives with the optimal use of resources. Thus it is possible to be 
effective without also being efficient, but it is not possible to be efficient 
without also being effective. 

Building on this basic distinction, we have found the following quotations 
useful in further clarifying terms: 

A more effective institution is typically defined as one whose students 
make greater progress over time than comparable students in comparable 
institutions. (John Gray, see Chapter 4 ) 
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Ifs difficult to concentrate on effectiveness when efficiency is so 
dominant. (Questionnaire response from a college principal) 

School improvement is about raising student achievement through 
enhancing the teaching learning process and the conditions that 
support it. It is about strategies for improving the school's capacity for 
providing quality education. (Hopkins, Ainscow and West, 1994) 

Quality. The delivery of a professional service in a consistent fashion to 
predefined standards and specifications which meet customer wants 
and needs and provide value for money. (Sallis and Hingley, 1992) 

Continuous improvement - the involvement of everyone in process 
improvement, now and forever. (Choppin, 1997) 

These definitions of effectiveness, improvement and quality demonstrate 
the problematic relationship between theory and practice. Researchers are 
tentative about ascribing specific causal relationships between critical 
variables and outcomes. There are always indeterminate factors that are 
difficult to identify, let alone quantify. One can only be guided by the best 
available evidence. 

In practice, we found that there is often confusion about the definition of 
effectiveness. For many college staff, the emphasis in recent years on reducing 
costs has made effectiveness more or less synonymous with efficiency. This is 
a confusion. Effective institutions may not be cheap to run, although value 
for money is obviously an important additional factor. 

Neither is an improving school the same as an effective school. Much of the 
US and UK effectiveness literature focuses on factors associated with effective 
schools (Levine and Lezotte, 1990; Sammons et al, 1995). However, as 
Austin and Reynolds (1990) point out: 

We at present know more about the characteristics of effective schools 
than we know about the precise ways in which schools can be made 
more effective. 

Scheerens (1992) switches our attention from institutional to teacher level. 
Drawing on research undertaken in the Netherlands, he focuses on effec- 
tiveness factors at the critical instructional level. At this level, effectiveness 
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depends on the interrelationship of such factors as time available for learning, 
qualitative components of the instructional process, the instructional 
material, grouping and teaching behaviour. 

In terms of improving effectiveness, focusing on the effectiveness of the 
teacher, rather than effectiveness at institutional level, makes it somewhat 
easier to relate behaviours to outcomes. The process of teaching can be seen 
to be the focal point for improvement and change (Hargreaves and 
Hopkins, 1991). However, the notion of continuous improvement in 
relation to students’ learning makes it difficult to reach a common definition 
of quality; what is recognised as quality now will, by definition, not be 
quality tomorrow. This presumes a view of quality as directly related to the 
experience of the learner, or the quality of the student experience. Systematic 
attention to changing the learning conditions as a key focus for improvement 
recognises the need for planned and systematic approaches to change as a 
process taking place over time. Thus, continuous improvement of quality 
becomes the hallmark of effectiveness. 

An understanding of how the terms are interpreted, applied and evaluated is 
crucial. Particular assumptions and ascribed meanings influence the very 
nature and use of the data. 

The research design that is required to study school improvement from 
a school effectiveness perspective is quite difficult and time-consuming 
to construct and implement ... [It bears] out the old adage that the 
data one has tends to structure the way one sees the problems, and the 
way one sees the problems tends to structure the data one attempts to 
collect. (Gray etal., 1993) 

Key points 

* There is confusion about the terms ‘effectiveness’ and ‘improvement’. 

® Efficiency is not the same as effectiveness. The former is concerned with the 
relationship between inputs and outputs, whereas the latter is concerned 
with whether the objectives or purposes of the institution have been 
achieved. Improvement implies change over time. 
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• An improving institution may not be an effective institution, and effective 
institutions may not be cheap or efficient. 

• The focus for improving effectiveness may be best directed to the level of 
the teacher, where it may be relatively easier to relate outcomes to teaching 
behaviour. 

• However, there has to be a generally agreed definition of quality; this can 
change over time. 
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School effectiveness 
research 



This chapter sets out the evidence from research about ways of identifying 
effective educational institutions and those that are improving, as well as the 
likely characteristics of effective educational institutions. There is an 
extensive body of research relating to the schools sector within the specific 
research paradigm known as SER. There is almost no comparable research 
relating to further education. However, a diffuse body of literature touches 
on educational effectiveness and improvement issues in relation to further 
education. 

The defining features of SER are that: 

• it is concerned with the outcomes of education 

• institutions should be compared like with like 

• processes are mainly of interest insofar as they 
are related to outcomes. 

Much of the research within this paradigm has been of an essentially 
quantitative nature, but more recent qualitative studies have also made 
influential contributions. 
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Definitions of effectiveness 



A more effective institution is typically defined as one whose students make 
greater progress over time than comparable students in comparable institutions. 
Attending a more effective school, as opposed to a less effective one, seems 
to make up to eight to ten points difference to a student’s overall exam 
points score (where seven is given for an A grade, six for a B, down to one for 
a G). In other words, a more effective school adds up two grade C passes 
more at GCSE per pupil than a less effective one (Gray and Wilcox, 1995). 
Higher figures are sometimes reported, but one needs to be sensitive to 
the nature of the comparisons, which have sometimes involved schools at 
different extremes. 

Nearly all the secondary school studies use exam results as the outcome 
measure. These tend to be highly correlated with each other (0.85 and 
above). The measures in public use (such as five A-Cs at GCSE and two or 
more A-levels) may be convenient, but tend to have poor statistical charac- 
teristics. They may also encourage very focused behaviour on improving 
the results of ‘borderline’ students. Measures that reward effort equally 
anywhere along the scale are preferable (e.g. a rise from E to D is valued at 
one point, and a rise from D to C also scores one point). 

Other outcome measures - such as students’ attitudes toward their education, 
attendance and truancy rates, and post-school destinations - have also 
sometimes been used. The different measures are undoubtedly correlated 
(especially those relating to exam performance), suggesting that there is such 
a thing as a more effective school rather than schools that are more effective 
at this particular activity or that. 

Rather than rely solely upon exam results, different researchers have identified 
a wide range of measures that could be used in addition. Often too many 
indicators are used. To counteract this. Gray and Wilcox (1995) suggest that 
just three performance indicators are capable of capturing all the most 
important components of a school’s work. These are the proportions of 
students who: 

• make more academic progress than expected 

• are satisfied with the education they are receiving 

• have a good or ‘vital’ relationship with one or more 
adults/teachers in their school. 
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In choosing such indicators, one needs to bear in mind the importance of 
‘stretching’ institutions in different directions, rather than simply piling 
them up and measuring roughly the same set of institutional attributes. 

The importance of comparing like with like 

School effectiveness studies have the premise that like is being compared 
with like. There is now fairly widespread agreement that the most appropriate 
control for differences in schools’ intakes is connected with individual students’ 
prior attainments at entry. For example, students’ prior attainments at GCSE 
can be used for studies of 16-19 institutions (see below). 

Information about prior performance is not always available. In these 
circumstances, other information about pupils’ backgrounds is usually sub- 
stituted, such as eligibility for free school meals. Drawing on census data, it 
is possible to establish the general characteristics of students in the area of 
the school. This makes it possible to construct ‘families’ of like schools. 

How much information is needed depends, in practice, on what is available. 
With one or two ‘good’ controls (such as prior attainment), only a few 
additional items of information (such as students’ gender and ethnic background) 
may be needed to capture most of the variation in student performance that 
can potentially be attributed to prior/background characteristics. Where such 
measures are not available, however, rather more information may be required 
to satisfy the expectation that like is truly being compared with like. 

Comparisons of institutional performance 

School effectiveness researchers believe that, while some useful things can be 
said about institutions’ effectiveness, there is a danger of over-interpreting 
the data. It is almost certainly inappropriate to divide institutions into more 
than three groups: 

• those of average effectiveness, performing around the levels that 
would be predicted from knowledge of their intakes 

• those that are more' effective 
® those that are less effective. 



O 



3 ^ 



V0I2 No 12 



fed a report 



The groupings are determined by a statistical procedure that establishes 
whether the results for individual institutions depart sufficiently from the 
average to be of statistical significance. Usually between two-thirds and 
three-quarters of institutions are found to be of average effectiveness; the 
remaining third to a quarter are divided fairly equally between the more and 
less effective. In one such study, Gray et al. (1995) found evidence that some 
institutions did better with some kinds of candidates than others, and that 
the kind of institution candidates attended made some difference to the 
prediction of their examination performance (see below). 



Further issues 

A recent study by Sammons, Thomas and Mortimore (1997) has shown that 
much of the variation in schools’ effectiveness can best be explained in terms 
of differences between departments. In practice, very few schools appear to 
be equally effective across the board. Few, for example, perform equally well 
in English, Maths and Science; most have a mixture of stronger and weaker 
departments. 

Most studies of school effectiveness have been based on single cohorts of 
students and have not usually been replicated. In general, such studies have 
found that schools that do well one year tend to do well in the subsequent 
year. (See Gray and Wilcox, 1995 for a fuller review.) More recently, however, 
the implications of stability for the study of school improvement have begun 
to be considered; if there is too much stability from year to year then the 
element of change needed for institutional improvement is ruled out. 



Factors that seem to make a difference 

What makes one school more effective than another? This is not easy to 
determine. Scheerens (1992) provides a review of the international literature 
and concludes that very few variables indeed have received what he terms 
‘multiple empirical research confirmation’. By this he means that they have 
been found to be of importance in several studies (not just one) in explaining 
variations in student outcomes. Only two (the amount of ‘structured teaching’ 
pupils receive and ‘effective learning time’) survive this stringent criterion. 
Some additional measures are judged to have ‘a reasonable empirical base’, 
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Table i Eleven factors associated with school effectiveness 


1 


Participatory leadership 


• firm and purposeful 

• a participative approach 

• the leading professional 


2 


Shared vision and goals 


• unity of purpose 

• consistent practice 

• collegiality and collaboration 


3 


A learning environment 


• an orderly atmosphere 

• an attractive working environment 


4 


Emphasis on teaching 
and learning 


• maximum learning time 

• academic emphasis 

• achievement focus 


5 


Purposeful teaching 


• efficient organisation 

• clarity of purpose 

• structured lessons 

• adaptive practice 


6 


High expectations 


• high expectations for all 

• expectations communicated 
® intellectual challenge 


7 


Positive reinforcement 


• clear and fair discipline 

• feedback 


8 


Monitoring and enquiry 


• monitoring pupil performance 

• evaluating school performance 


9 


Pupil rights and 
responsibilities 


• high pupil self-esteem 

• positions of responsibility 

• control of work 


10 


Home-school partnership 


• parental involvement in their 
children’s learning 


11 


Learning for all 


• school-based staff development 


Source: Sammons, Hillman and Mortimore ( 1994 ) 
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including pupils’ ‘opportunity to learn’, the ‘pressure to achieve’, the existence 
of ‘high expectations’, the extent of ‘parental involvement’. These views 
have subsequently been updated by Scheerens and Bosker (1997). 

A recent review of the international literature by Sammons, Hillman and 
Mortimore (1995) for OFSTED has been particularly influential. This identifies 
11 factors (see Table 1, page 33). Some of these factors, however, are better 
supported than others. The general view is that they are a guide to good 
practice rather than a blueprint for it. Seven out of ten schools that do these 
sorts of things tend to get better results, but this is not inevitably the case. 
Critics have argued that some of these factors are ‘obvious’ and could have 
been deduced by other means (see, for example. White and Barber, 1997). 
Nonetheless, they do seem to have some empirical support. 

A more humanistic perspective on what makes the difference is provided in 
Lightfoot’s (1983) study of American high schools, based on case studies of 
six ‘excellent’ institutions (see Table 2). Relationships between teachers and 
students feature much more prominently in this study than in the others, 
which are more managerial and organisational in orientation. 



Table 2 Seven characteristics of ‘goodness* in American high schools 


1 


Conscious of their imperfections; willing to search for 
their origins and solutions. 


2 


Development of a visible and explicit ideology that involves 
both staff and pupils. 


3 


The headteacher is seen as the voice of the school providing vision. 


4 


The senior management hold teachers and their work in high regard - 
their satisfaction and nurturing is seen as critical. 


5 


There is an easy rapport between teachers and pupils; 
teachers are not ‘afraid’ of pupils. 


6 


There is a visible concern for the weakest members of the institution. 


7 


Opportunities are fostered for pupils to make a ‘vital’ 
relationship with one (or more) adults. 


Source: Lightfoot (1983) 
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Developing management and teaching 

Recently, the role of the headteacher and, in particular, the leadership styles 
of senior managers, have attracted particular attention in SER. No single 
leadership style seems to be universally appropriate. More effective schools, 
however, seem to have management teams that offer a clear sense of mission 
to their colleagues and a capacity to translate their visions into a series of 
goals and specific targets. Their headteachers command colleagues’ respect 
even if they are not always popular. At the same time, effective management 
teams seem to have taken a variety of steps to ensure the active involvement of 
staff in planning the means to achieve the goals, through participatory 
decision-making processes. It is increasingly recognised that ‘leadership’ is 
required at other levels in the school, especially at departmental and 
classroom levels. The effective school, then, seems to have secured a balance 
between vertical push and horizontal pull, and between laterality and 
centralisation. Such factors seem to be especially important when the school 
considers the steps it needs to take to secure further improvements. 

Recent statistical analyses have shown how variations in performance can 
be partitioned between the different levels of the school (Sammons et al, 
1997). They have shown, in particular, the extent of variation between 
different classrooms in particular schools. In turn, these findings have 
fuelled interest in factors associated with effective teaching and some of the 
strategies that might help to support it. Again, as with leadership styles, no 
single approach has emerged as universally more effective (Cooper and 
McIntyre, 1996). There is now, however, greater awareness about the need 
for schools to develop a repertoire of strategies for supporting teaching 
(including a renewed interest in the potential of self-evaluation). 

From school effectiveness to school improvement 

What is meant by school improvement? There is increasing interest in 
definitions that are related, in some way, to students’ achievement. 
Borrowing from the school effectiveness tradition, an ‘improving’ institution 
is therefore one that ‘improves in its effectiveness over time’. To realise this 
kind of definition empirically, data on at least three cohorts of students 
passing through the system is required. 
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In recent years, researchers have become increasingly interested in how 
school improvement occurs and which features of schools provide the key 
levers for change (see Stoll and Fink, 1996, for a comprehensive review). 

One of the main reasons for this shift of perspective has been the realisation 
that the literature on school effectiveness factors may have been too static in 
its focus. What a school does to become more effective may differ from what 
it does to sustain effectiveness once it is achieved. Unfortunately, hardly any 
studies have looked at how schools have become more effective over time. 

Researchers are currently addressing a range of issues. Which particular features 
of schools, for example, provide key levers for change? Are some features of 
schools easier to change than others? And are the things that a school does to 
get into the pack the same as, or different from, those that help it to pull 
ahead? The change process has turned out to be rather more difficult to 
study than the correlates of school effectiveness. In the process researchers 
have also come to realise that they need a more elaborate framework, in 
which schools’ levels of effectiveness are integrated with their ‘improvement 
trajectories’. 

When schools start to change, they seem to have worked on any or all of the 
following things: 

• attitude and approach to planning 

• how the school is run and organised 

• the way the curriculum is organised 

• the ethos or culture of the school 

• aspects of the quality of teaching and learning. 

Indeed, they often launch change on several of these fronts at the same time. 

In the recently completed Improving schools study, the schools that improved 
most rapidly focused on two things simultaneously (Gray et aL, in press). 
They adopted a series of ‘tactics’ in relation to improving exam performance; 
and they tried, at the same time, to facilitate discussions and work about the 
quality of teaching and learning in various subject departments. By contrast, 
schools that were less effective and improving only slowly, at best, spent a lot 
of time getting to the starting gate and dealing with ‘cultures of resistance’. 
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Key points 

• There is an extensive body of research relating to the schools sector within 
the specific research paradigm known as SER. 

• The defining features of SER are as follows: 

• it is concerned with with the outcomes of education; the most 
popular measures are examination results, but a range of other 
performance indicators have been developed 

• institutions should be compared like with like; this is usually 
achieved by comparing schools’ intakes, e.g. individual students’ 
prior attainments at entry 

• processes are mainly of interest insofar as they are related 
to outcomes. 

• Several studies have identified factors that seem to make a difference. Those 
presented by Sammons, Hillman and Mortimore (1994) and Lightfoot 
(1982) are of particular interest. 

• There is increasing interest in research into how schools improve. Schools 
that start to change have usually worked on: 

• the school’s attitude and approach to planning 

• the way the school is run and organised 

• the way the curriculum is organised 

• the ethos or culture of the school 

• aspects of the quality of teaching and learning. 
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Research on issues 
relating to college 
effectiveness in 
further education 




Overview 

Relatively few publications deal with issues of quality, effectiveness and 
improvement specifically in relation to further education. An important 
early text is Theodossin’s (1985) In Search of the Responsive College. It provides 
a concise overview of key factors affecting colleges before incorporation. 
Taking inspiration from the American management classic In Search of 
Excellence (Peters and Waterman, 1982), Theodossin initially planned to 
identify excellent innovative colleges with the aim of promoting dissemination 
of their characteristics across the FE system. Finding it impossible to create 
an authoritative Ust of excellent/responsive colleges, he settled on a presentation, 
based on his respondents’ answers, of current thinking on the concept of 
responsiveness in relation to colleges. 

Theodossin identified a tension in the system between traditional educational 
values (which he characterised as ‘missionary’ in nature), and those of 
industry/commerce (held by the ‘marketeers’). It was felt that a responsive 
college would put client needs before provider needs, and it was evident that 
this growing perception was generating some anxiety in colleges. There was 
a strong feeling that improved market research was going to be crucial to 
future success. Theodossin also found that formal systematic quality control 
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data collection was generally absent and concluded that, in an increasingly 
competitive environment, . performance indicators and output measures 
may come to be recognised as a hitherto ignored source of ammunition.’ 

John Stone’s (1997) Increasing Effectiveness: A Guide to Quality Management 
has, despite having been pubUshed prior to the FEFC’s (1997) new requirements 
for inspection, been credited with providing: 

...a very useful framework for colleges who are in the throes of 
preparing their own systems of self assessment and and assuring that they 
have a robust system of quality assurance in place. (Parnham, 1998) 

Stone outlines a range of possible approaches to the achievement of quality, 
drawing on his experience in the creation of quality systems at Swindon 
College. He provides definitions and overviews of the concept of quality, 
external quality awards, and of TQM, and then discusses issues arising from 
the implementation and management of quality control mechanisms. 
Containing checklists and examples of standard forms, this is a sound 
introductory text for those who are looking for a basic overview of the key 
principles of establishing and working with quality control systems. 

The Lecturer’s Guide to Quality and Standards in Colleges and Universities 
(Ashcroft and Foreman-Peck, 1995) is a practical handbook incorporating a 
range of ‘enquiry tasks’ designed to facilitate the exploration of issues in the 
reader’s own workplace. Covering processes in quality and standards, and 
examining quality issues in relation to teaching and learning, student 
support, staffing and staff development, assessment and evaluation, course 
design, resource management, marketing and recruitment, and research, the 
book is relatively comprehensive. Focusing on the perspective of the lecturer, 
and employing a reflective practice model, it constitutes a useful staff 
development/quality consciousness-raising resource. 

500 Tips for Quality Enhancement in Universities and Colleges (Brown, 
Race and Smith, 1997) provides a pragmatic approach to improvement. 
This book consists of a series of brief items of advice organised into broad 
categories, including valuing students, teaching and assessment, quality 
processes, preparation for quality assessment visits, and caring for the campus. 
The format facilitates quick reference and generates ideas, with a focus on 
quality enhancement rather than quality assurance. 
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A recent and useful collection of analytical papers focusing on the relationship 
between effectiveness and improvement in schools and colleges is contained 
in Organisational Effectiveness and Improvement in Education (Harris, 
Bennett, and Preedy, 1997a). Based on the premise that effective management is 
. . central to the professional development of all teachers and lecturers . . 
the 25 papers featured present a range of perspectives to make management 
theory accessible. Although the balance of the papers veers toward schools, 
the issues explored apply across the sectors. The book provides a stimulating 
exploration of systems theories, the concepts of effectiveness and improvement, 
educational cultures, value added and change management. It offers a 
relevant insight into the wider literature on organisation theory. Several 
contributors articulate a multi-level conception of organisational effectiveness, 
incorporating the student, the classroom and institutional levels, and 
arguing the need for these to be dynamically interrelated. 

Beyond the few books on college effectiveness lie the many papers published 
in academic journals. A highly relevant journal is Quality Assurance in 
Education (published by MCB University Press). Critically examining the 
issues surrounding quality in education, this journal seeks to feature papers 
from a range of stakeholders, both internal and external to education, to 
encourage dialogue around the meaning of ‘quality’ and to inform both 
practice and policy. 

FE colleges have, of course, produced their own unpublished papers and 
documentation that are the basis of their management and systems. In 
addition to this ‘functional documentation’, colleges have increasingly 
undertaken - individually and collectively - research projects that have 
contributed directly to improving practice. For a recent overview of research 
activity in FE colleges, together with 10 case studies, see Johnson (1997). 
Johnson’s survey of 150 colleges showed that 66% were involved in research 
to improve institutional performance or quality, the second highest area of 
research activity (behind marketing at 93%). 

Although this study reviewed research over the past 25 years, this report is 
generally limited to studies produced since colleges became independent 
corporations in April 1993. The establishment of the FEFC and the new 
inspection regime, and the rapid expansion of the sector, have brought about 
very significant changes since that date, so that earlier studies inevitably have 
less relevance. 
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Two studies in the school effectiveness paradigm 

Our literature search uncovered only a handful of studies into college 
effectiveness in further education within the paradigm of SER reviewed 
above. We will discuss two of these here: the study carried out by Gray in 
collaboration with DfEE statisticians (Gray et al.^ 1995), and the unpublished 
study by Conway of the achievements of students in the sixth-form college 
of which he is the principal. (For an account of the system, see Conway, 
1997.) A detailed analysis of Gray’s study is included as an example of the 
kind of analysis that is produced by this kind of research. 

Value added monitoring, when embedded into student support and MIS, 
has been shown to offer substantial pedagogic and learning potential in 
relation to A-level programmes. However, it should be recognised that the 
diversity of the EE client group, together with the assessment regimes 
currently associated with the various vocational curricula, have made its 
broad application problematic (see FEU, 1993 and 1994; FEU/FED A, 1995; 
Barnard and Dixon, 1998). 

The development of a national framework for estimating 
value added at GCE A/AS-level 

The study by Gray et al. (1995) compares the institutional performance of 
16-19 institutions. It can be said to compare like with like, because only 
students studying for A-levels were included. The data set used for this analysis 
covered virtually every post- 16 institution in the country. It used students’ 
A-level scores as the outcome measure and their prior performance at GCSE 
to account for their attainment at entry. Similar work has been undertaken 
using the A Level Information System (ALIS) data sets (Fitz-Gibbon, 1996). 

Figure 1 (opposite) is drawn from the DfEE study (Gray et al., 1995). It 
shows the relationships between pupils’ performance at A/AS-level and their 
performance at GCSE, as well as other information about their background 
characteristics. Overall, there would seem to be a strong relationship 
between a student’s performance at GCSE and subsequent performance at 
A/AS-level; the higher the performance at GCSE, the higher the performance 
at A/AS-level. The figure also shows the predicted results for five different 
institutions. 
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The lines on the figure show the results for the typical institution nationally 
(labelled C) as well as typical institutions at varying levels of ‘effectiveness’, 
including institutions just falling into the top and bottom deciles of 
effectiveness (A and E) and at the upper and lower quartiles (B and D). Only 
a relatively small proportion of institutions were significantly different from 
the national norm (about 28 % of all institutions); these were fairly evenly 
divided between those that were doing considerably better than predicted 
and those that were doing considerably worse. 



Figure i Estimates of A/ AS-level scores for candidates with different ‘average 
GCSE grades’ broken down by ‘effectiveness’ of their institution 




Key: Institutions — A B C D E 
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There was evidence from the analyses that some institutions did better with 
some kinds of candidates (girls or higher-attaining students, for example) 
than others. This phenomenon is known as differential effectiveness and is 
shown in Figure 2 (below). Institution A was one of those with the ‘steepest’ 
slopes ; it boosted the performance of higher-attaining candidates relative to 
lower-attaining ones. By contrast, institution E was one of those with the 
‘flattest’ slopes; it tended to boost the performance of lower-attaining candidates 
relative to higher-attaining ones. The analysis suggested that about 11% of 
all institutions were differentially effective with respect to candidates’ prior 
attainments; again roughly half of this group tended to do better with the 
more able candidates, while half did better with the less able ones. 
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Figure 3 Estimates of A/AS-level scores for institutions with differing missions 




Key: GCE A/AS-level main focus (selective) 

Focus mainly on GCE A/AS-level 

= — GCE A/AS-level vocational courses 



The same analysis showed that the kind of institution candidates attended 
made some difference to the prediction of their A/AS-level performance (see 
Figure 3, above). Institutions whose main mission or focus was on A/AS- 
level exams, and that were able to select their students, did slightly better in 
terms of ‘effectiveness’ than other institutions that were mainly concerned 
with A/AS-levels and not able (for whatever reason) to select their students. 
Finally, both of these groups did slightly better than the third group, which 
combined preparing students for A/AS-level courses with preparation, at the 
same time, for more vocational courses. Interestingly, this third group 
(which included most FE colleges) tended to favour candidates with lower 
average GCSE grades compared with higher-attaining candidates. 
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Consideration of prior attainment accounts for a considerable proportion of the 
variation in students’ performances. Nonetheless, some differences between 
institutions remain. The evidence in Figure 1 suggests that a candidate 
attending one of the institutions in the top 10% of effectiveness (Institution A) 
would secure nearly three more A/AS-level points than a candidate 
attending the typical national institution (labelled C). The latter would, in 
turn, secure just over two points more than a candidate attending an institution 
in the bottom 10% of effectiveness. The range between the top 10% and the 
bottom 10% of institutions in terms of their effectiveness is equivalent to 
about five points, or two and a half grades at A-level. (A difference of two 
points is equivalent to one grade at A-level.) 

Quality is not a certificate: it is about 
continuous improvement everyday 

This study by Conway (1997) adopts a similar approach, but the analysis 
is much simpler. Five 16-19 institutions were compared, representing 
different ‘types’ of colleges categorised as follows: 

• creamed comprehensive 

• good fully comprehensive 

• selective grammar school 

• sixth-form college 

• top independent school. 

The Universities and Colleges Admissions Service (UCAS) scores for all students 
were analysed in relation to their GCSE points prior to commencing the 
course, using the standard points allocations: 

GCSE: A=5, B=3, C=1 

UCAS: A-level A=10, B=8, C=6, D=4, E=2 

UCAS: AS-level A=5, B=4, C=3, D=2, E=l. 

Analysis showed that GCSE points were a consistently good predictor of 
A-level performance, regardless of which kind of institution the student 
attended. This provided a strong indication that all the institutions were 
equally effective despite their very different intakes and institutional cultures. 
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Conway was also able to use this data to identify an individual subject in 
which students were underperforming at A-level, and to take action to ensure 
improvement in student achievement in this subject in the following year. 

In a working document, Conway (1997) describes the ‘stunning’ effects of 
putting his ideas into practice: the UCAS points score per student rose from 
11.9 points in 1987 to 24.9 points in 1997. The analysis of A-level results is 
matched by an emphasis on valuing each student as an individual. Conway 
believes that his system can be used to assess value added on a national basis: 

The scheme outlined is now tried, tested and simple. It is a practical 
process which values young human beings in much more than examination 
results ...It is powerful as an analytical tool for those managing schools. 



One FEDA study relates specifically to self-assessment, which has since 
become integral to the FEFC inspection process. The focus on college self- 
assessment of quality and performance arose from the recognition that such 
procedures were fundamental to the dynamic of improvement, and from the 
sense that many ‘. . . would equate the ability to self-assess with the maturity 
of the FE sector’ (Dixon, 1996). This study suggested that self-assessment 
was a key component of quality improvement. It pointed out, however, that 
many colleges had not yet developed a culture of judging quality performance, 
and this would be necessary if they were to become the ‘. . . self-critical 
institutions which set and achieve high standards . . .’ referred to by the 
FEFC’s Chief Inspector {Times Educational Supplement, 1996). 



Among the most cited modern management texts are the Peters and 
Waterman classic In Search of Excellence (1982), and Peters’ follow-up 
Thriving on Chaos (1989). The essential message of these texts is the need 
for organisations to focus on establishing an appropriate culture. Changing 
the Culture of a College (Gorringe and Toogood, 1994) provides eight 
accounts of change in different colleges, each written by a nationally 
prominent principal, director or chief executive. This collection is useful in 



Self-assessment of quality and performance 



Changingthe culture of the college 
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drawing together diverse views grounded in the management of change in 
colleges. There is no overarching theme or theory and, although the individual 
pieces are relatively short, each successfully evokes distinctive visions of 
college cultures and how to build them. 

Ralph (1995), through an action learning study, provides a detailed and 
theoretically sophisticated account of innovation and change processes as 
experienced by change agents in a single college. This impressive contribution 
suggests, significantly in the context of a general FE climate influenced by 
classical industrial quality regimes, that: 

The learning organisation develops out of the praxis and growth of its 

members and its promotion does not require a ‘right first time’ culture. 

Levacic and Clatter (1997) have edited a collection of papers focusing on 
change management issues in further education in England, Scotland and Wales. 
The five studies featured focus respectively on: the development of strategic 
management in three colleges; strategic responses to pressures of competition 
and cost-cutting; learning strategies adopted at three colleges; subjective 
views of change held by staff at different levels in five colleges; and cooperative 
partnerships between schools, colleges and universities (through nine case 
studies). The collection as a whole outlines a series of responses to the post- 
incorporation climate, including the impulse to create more learner-centred 
courses, more innovative marketing, improved administration and student 
support, and upgraded physical environments. Tensions arising from change 
are not ignored, and the collection as a whole concisely traverses the wide 
territory demanded by a study of management of change in further education. 

Management, leadership and 
organisational development 

Three studies of leadership and organisational development are worth high- 
lighting. Harper (1997) suggests that being a manager in a college is essentially 
the same as being a manager in other organisations. Certainly management 
may be regarded as a generic skill rather than as a specific occupation, and 
FE colleges are invariably substantial organisations with large budgets, 
significant numbers of staff, costly physical infrastructures, and challenging 
missions to accomplish. They are organisations for learning as well as 
learning organisations. 



48 



fed a report 



Vol2 No 12 



Jones (1994) has argued that hierarchical, class-based attitudes have prevented 
most British organisations from changing their cultures to the more successful 
and fundamentally more democratic ones evident in many competitor 
economies. According to Jones, these entrenched industrial attitudes are 
also prevalent in the education sector. It is suggested that: 

. . . for all the rhetoric over recent years concerning quality, empowerment, 
TQM, customer care etc., most British organisations in education and 
industry show a fundamental misunderstanding of what is required for 
effective action, and continue in hierarchical working, teaching and 
learning practices. 

Crawford (1997), suggests: 

All teams call for effective leading and following at every level. Thus 
leading is not a one-dimensional activity but a process in which more 
than one person is engaged, whether this is within a whole organisation or 
specific team setting. Proficient leading and team working is, therefore, 
central to effective performance within schools and colleges. 

Student retention 

The Audit Commission’s (1993) report Unfinished Business highlighted the 
existence of a ‘retention problem’. Many colleges have examined their 
internal mechanisms in relation to this phenomenon. Case study research 
(Martinez, 1995 and 1996) has found empirical support for some of the 
ideas from management of change theory: 

Ownership by and support from senior managers is important. Action 
in conditions of partial and incomplete information is preferable to an 
endless search for perfect information. All the case study strategies 
were driven by people who adopted what can be described as change 
agent roles . . . the successful strategies all embody elements of college 
transformation . . . 

More recent FEDA publications relating to the retention issue are Barwauh, 
Green and Lawson (1997), Martinez (1997), and Martinez and Munday 
(1998). Martinez, Houghton and Krupska (1998) focus specifically on the 
question of professional development for college staff in relation to the 
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retention strategies issue. Martinez and Munday (1998) is based on a survey 
involving more than 9,000 students and staff in 31 colleges in England and 
Wales. It is the largest British retention study undertaken in further edu- 
cation and clearly indicates the significant effect that the quality of service 
provided by a college can have on rates of student retention. The most sig- 
nificant reason for students leaving prior to completion was their belief that 
the course was not appropriate to their needs. This and earlier research has 
shown that reasons for student withdrawal are multi-causal, and not always 
easy to categorise. Nonetheless, colleges can, through the provision of 
appropriate guidance and support, try to solve this problem. 

A fine unpublished account of a collaborative research project on retention 
involving eight colleges and funded by North Yorkshire TEC is provided by 
Kenwright (1997). This study emphasised the need for advice and guidance 
procedures at entry, as well as the provision of on-programme support and 
tracking, including the use of targets. Earlier single college-based studies 
include Lambeth College (1994) and Stockport College (1995). 

Spours (1997) provides a review of research based on student retention in 
further education, pointing to a strong concern with student attitudes. Based 
on a case study of five colleges, Spours argues that there is a need for a close 
examination of staff attitudes on a range issues involved in retention. Spours 
is critical of the influence exerted by FEFC funding methodologies, and 
convincingly argues for the need for colleges to focus on achievement and 
progression, as opposed to retention, thereby placing educational missions 
and the professionalism of lecturers at the core of their practices. 



Curriculum 

In 1987 the Further Education Unit published its Quality in NAFE report on 
issues of quality in respect of work-related, non-advanced EE curricula 
(WRNAFE) for the Manpower Services Commission. FEU argued unequivocally 
that the key issue in quality was the curriculum, and that approaches to 
improving quality would therefore need to be curriculum led. The nature of 
the curricula deployed in colleges must necessarily have a fundamental effect 
on the quality of learning experienced by students. Smithers and Robinson 
(1993) points to the link between curricula and teacher effectiveness: 
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The ‘competence-based’ approach has important implications for colleges. 

It reflects a belief that demonstrable ‘competence’ is the whole aim, so 
that how the competence is acquired is strictly speaking irrelevant. It 
requires assessment to be available on demand. It shifts the focus from 
curriculum and education to assessment and qualification. It has been 
suggested that such an approach, taken to an extreme, threatens colleges 
with becoming the education equivalent of driving test centresl 

Bloomer (1997) has, through a substantial study of the experience of the 
curriculum by students and teachers, attacked the technocratic processes of 
curriculum reform in post- 16 education in the mid to late 1990s. Edwards 
et al. (1997) have examined the issue of ‘parity of esteem’ between vocational 
and academic curricula, pointing to an educationally damaging divide 
between the two. 

Barnard and Dixon (1998) have recently discussed the application of value 
added data in the context of further education. They demonstrated the utility 
of pedagogic devices such as target setting and ‘chance charts’. The latter 
enable students to see statistical information that can help them make an 
informed judgement regarding their individual likelihood of success should 
they make a particular subject choice. There is a need, however, for both 
effective MIS and for the careful use of the output data in the context of appro- 
priately sensitive tutorial support. Barnard and Dixon (1998) stress that this: 

. . . requires tutorial staff who are able both to offer the interpersonal 
skills needed to motivate students by negotiating realistic targets 
and agreeing effective courses of action, and to cope with elementary 
statistical concepts. 

Notwithstanding this proviso, the diversity of the FE client group, taken 
together with the relative complexity of the FE curriculum and the nature of 
assessment associated with much of it, combine to make the effective application 
of value added techniques in colleges problematic, certainly for the time 
being. Barnard and Dixon (1998), following a two-year study of nearly 
2,000 students in 10 colleges, state that: 

... it is not at present feasible to apply nationwide a single uniform 
methodology for the measurement of value added on vocational 
courses. [Nevertheless] . . . colleges should not be discouraged from 
using local value-added measurements, provided they are based on a 
sound statistical methodology. 
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Teaching and learning 

There is renewed interest in teaching and learning as the core component of 
college effectiveness. The emphasis in recent years has appeared to be more 
on structures, management practices and quality systems, but this is now 
changing. 

[W]e need constantly to remind ourselves of the essentially moral and 
social purpose — not only of education but also of training. Jerome 
Bruner’s three questions ‘What makes people human?’, ‘How did they 
become so?’, ‘How might they be more so?’ have sadly become instead 
‘What makes people wealthy?’, ‘How do they become so?’ and ‘How 
might they be more so?’ (Crombie White, Pring, and Brockington, 1995) 

Cunningham (1997) has addressed the issue of problems experienced and 
caused by ineffective teachers in post-compulsory education. The highly 
competitive and monitored environment of further education is a context in 
which scrutiny of individual performance has been intensified. Charters have 
created a situation where it is the norm for poor performance to be reported. 
Monitoring of student retention can often lead directly to problem tutors. 
The 1996-97 FEFC Chief Inspector’s annual report (FEFC, 1997d) indi- 
cated that strengths outweighed weaknesses in 61% of classes observed. 
Between 1993 and 1997 an average annual 8% of sessions observed had 
weaknesses that outweighed strengths. Most classes observed by inspectors 
then were deemed to be ‘well planned and effective’. 

Cunningham suggests that remedies for the problems facing failing teachers 
might include provision of support through peer observation and mentor- 
ship, together with the encouragement of reflective practice, through the 
maintenance of a diary or critical incident analysis. 

Ashcroft and Foreman-Peck (1996) have perceptively observed that: 

The process of framing professional and pedagogic development in 
terms of performance indicators and quality standards may be necessary 
to satisfy the need for accountability to outside institutions. The ability 
to describe action in these terms is a management skill, they [models of 
quality] may oversimplify reality, and an uncritical adherence can 
distort other aspects of quality. Reflective practitioners do not see 
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themselves as solely accountable to those with power, but also to those, 
including themselves, who have an interest in the educational process 
but limited opportunities to influence its overall direction. 

It is important that lecturers in further education are able to think of themselves 
as professionals. Elliott (1998) has persuasively made the connection between 
the working practices of lecturers and their sense of the value and worth of 
what they do. This underlines the necessity of recognising the centrality of 
processes of teaching and learning in questions of college effectiveness, and the 
crucial nature of the task of ensuring that these are given a higher profile than 
they have hitherto enjoyed in the debates surrounding quality in the sector. 

Equity, entitlement and inclusivity 

The landmark Tomlinson report has been dealt with in an earlier chapter. 
There are an increasing number of studies dealing with the broad range of equity 
issues relevant to the sector. Here we shall, of necessity, mention just a few. 

Equal Opportunities in Colleges and Universities: Towards Better Practices 
(Parish, McPake, Powney and Weiner, 1995) examines the effectiveness of 
equal opportunities initiatives, through case studies involving an FE college 
and two universities (one ‘new’ and one ‘old’). The book is primarily concerned 
with staff rather than students; however the detailed case studies and careful 
analysis of data offer some theoretically informed insights (micro-political, 
post-structuralist and feminist) that would benefit those involved in the 
design and implementation of equal opportunities policies. 

A persuasive rationale for an inclusive further education response to 
community needs, in the context of changes following incorporation, as well as 
in response to wider social trends, was first articulated by McGinty and Fish 
(1993). Choice involves knowing what options are available: FEU (1994) 
underlined the direct connection between the delivery of proper guidance to 
college students and the quality of their experience. Real choice in society 
also requires that individuals feel empowered to make and act on their own 
decisions. This often means that underprivilege and discrimination must be 
overcome. The views of some students with disabilities or learning difficulties 
may be found in Student Voices (Skill/SCPR/FEFC, 1996). 
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Franklin (1992) described how East Birmingham College used the notion of 
quality to advance education for a multicultural society, by making the rig- 
orous pursuit of the college’s equal opportunities policy an important part of 
the philosophy enshrined in its quality policy. Gray, Jesson and Tranmer’s 
(1993) youth cohort study for England and Wales showed that more 
females stay in full-time post- 16 education than males. It was also the case 
that young black people were ‘. . . at least as likely (and often more likely) to 
stay on post-16 than their white counterparts.’ These facts may be partially 
explained by labour markets, but it seems that questions of access to educational 
opportunity for women and blacks may not be very different from those 
that apply to other groups. The particular ways in which education may be 
experienced by members of disadvantaged social categories is a more 
complex matter. 

Stott and Lawson (1997) reported on a survey of 22 colleges conducted by 
the Network of Women Managers, which revealed that women constituted 
83% of the student body, 58% of all employees, 38% of middle managers, 
and 27% of principals. Their study examines the strategies used by women 
principals to manage their work and thereby makes an important contribution 
to research on gender issues in further education. 

Published in support of Learning Works, a useful volume of statistical 
information (FEFC, 1997f) incorporated a discussion of how to identify EE 
students requiring additional resources. The three options considered for 
this were: 

• the social and economic characteristics of where students lived, 
as identified by postcode 

• income level, as indicated by entitlement to benefit 

• previous educational achievements. 

It concluded that the latter provides the best option, but that data collection 
in this regard is currently a problem; until this can be remedied, postcodes 
will be used. A separate report. How to Widen Participation: A guide to 
good practice (FEFC, 1997g) provides a framework by which colleges can 
measure their performance in widening participation. 
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Key points 



Gray etal. (1995) compare the institutional performance of 16-19 institutions 
and show that: 

• about 1 1 % of institutions appear to make a difference, about 
half of these doing better than average and about half less well 

• the kind of institution does appear to have some effect on 
students’ examination performance. 

Conway (1997) describes how: 

• GCSE scores can be used to predict students 
performance at A-level 

• these data can be used to improve the work 
of underperforming departments 

® the approach can be combined with valuing 
each student as an individual. 

A diffuse body of literature touches on educational effectiveness and 
improvement issues in relation to further education. These studies suggest that: 

• self-assessment is a key component of quality improvement, 
but it depends on college culture 

• leadership and organisational development are important 
factors in college effectiveness 

® developing the culture of a learning organisation depends on allowing 
staff to grow, and is different from a ‘right first time’ culture 

® we now understand retention issues better and many colleges 
are succeeding in improving their retention rates 

® further work needs to be done on developing the curriculum, 
particularly relating to the issue of parity of esteem between 
A-levels and GNVQs 

® a single value-added methodology cannot currently be 
applied to further education 

® there is renewed interest in teaching and learning as the core 
component of college effectiveness. The emphasis in recent years 
has appeared to be more on structures, management practices 
and quality systems, but this is now changing. 
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Systems for inspection 
and improvement 



This chapter outlines features of current policy initiatives and regimes for 
inspection in the FE and schools sectors which are designed to improve 
institutional effectiveness. While there are recognisable similarities between 
the policies and inspection regimes for each sector, there are also distinct 
differences. The sectors are, therefore, dealt with separately to enable us to: 

• draw out messages from the schools sector that may be useful for 
further education 

• clarify the current background for quality improvement work in 
the FE sector to make appropriate recommendations for future 
research and development work. 



Further education 

The inspection regime established by the FEFC provides the framework for 
any quality improvement work that will be carried out in the FE sector in the 
immediate future. Inspection is a legal duty required of the FEFC and 
FEFCW by the Further and Higher Education Act (1992). 

Approaches to inspection in Wales and England are similar. In Wales, institu- 
tional effectiveness is being considered via two strands of activity at institutions: 

• the provision overall (educational effectiveness) 

• the governance and management of the institution 
(functional effectiveness). 
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The relatively comprehensive nature of such assessment recognises the 
potential effect on quality and standards of all aspects of an institution’s 
work. Acting on behalf of the FEFCW, and within the Council’s quality 
assessment framework, educational effectiveness will be assessed by the 
Office of Her Majesty’s Chief Inspector of Schools (Wales) (OHMCI(W)). 
FEFCW officers will undertake the assessment of functional effectiveness. 
Frameworks for both areas of assessment are being piloted during 1998-99 
at three colleges. 

The emphasis in Wales until recently has been on improving functional 
effectiveness. This is concerned with the good management, financial and 
governance practices in the colleges. Educational effectiveness could not be 
achieved until a sound and efficient management structure was established in 
the colleges. A framework for assessing educational effectiveness is now being 
developed, and will be piloted during the next academic year in three colleges. 

In developing the procedures for assessing institutional effectiveness, the 
emphasis in Wales is on college self-assessment and helping colleges to 
address any areas of weakness. Once the latter has been identified, an action 
plan will be produced. Each college will be assessed in relation to its own 
particular context, taking account of its mission statement. Comparisons 
with national standards will be made as appropriate. Good practice is 
expected to be disseminated among institutions. 

In England, a revised framework, drawn up following extensive consultation, 
was set out in Circular 97/12 (FEFC, 1997c). It replaced that which had 
been established by Assessing Achievement (FEFC, 1993a). The four-year 
cycle of inspection has been retained. The main purposes of inspection are 
now to: 

• validate college self-assessment 

• encourage continuous improvement of provision and raise standards 

• enable colleges to compare performance 

• assist the dissemination of good practice and identify nationally 
significant issues. 

Circular 97/12 states that the overriding aim of the inspection process is to 
‘promote improvements in quality through regular inspection and the publi- 
cation of inspection reports’. Colleges are also required to develop action 
plans as a result of inspection which now has a direct link with funding. 
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The inspection process incorporates college self-assessment, which is validated 
through FEFC inspection. Inspection principles detailed in Circular 97/12 
emphasise that this multi-dimensional approach to inspection is intended to 
raise standards and the quality of students’ experiences in sector colleges. 

Inspection under the new framework is more strongly focused on the 
curriculum. By this means, the FEFC is able to redirect the energy and 
attention of college staff toward the curriculum. Curriculum inspectors 
make an independent judgement about provision in relation to four key 
quality indicators: 

• teaching and learning 

• students’ achievement 

• curriculum organisation and management 

• staffing and specialist resources. 

Auditors from the inspection team award separate grades for governance 
and management. Curriculum inspectors provide evidence for these grades, 
as they do for the grading of cross-college services. 

Following consultation in 1997, the grade descriptors have been changed. 
They are now as follows: 

• grade 1 outstanding provision with many 
strengths and few weaknesses 

• grade 2 good provision in which the strengths 
clearly outweigh the weaknesses 

• grade 3 satisfactory provision with strengths 
but also some weaknesses 

• grade 4 less than satisfactory provision in which 
the weaknesses clearly outweigh the strengths 

• grade 5 poor provision with few strengths 
and many weaknesses. 

The self-critical culture is intended to lead directly to improvement. Circular 
97/13 (FEFC, 1997e) specifies that, to be effective, self-assessment must 
‘lead to actions which improve quality’. It makes clear that the most effective 
self-assessment reports (SARs) are concise and include references to quantifiable 
evidence that support judgements about strengths .and weaknesses. In 
addition, there are clear indications of deadlines for action, review dates and 
lines of responsibility. 
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In carrying out self-assessment reviews, colleges are directed to a range of 
areas intended to improve their effectiveness. These include, for example, 
‘the development of appropriate key skills’, the need to ‘improve student 
retention and achievement’, and better use of technology in teaching and 
learning. Circular 97/13 also draws attention to other quality assurance 
issues including the need to ensure the teaching skills of part-time teachers, 
and industrial updating opportunities for full-time teachers. The inspection 
process highlights the centrality of effective training, regular appraisal and 
appropriate forms of staff development for FE teachers as key indicators of 
college effectiveness. 

A crucial part of the self-assessment process is the improved flow and use of 
management information, analysed by computer. The FEFC will shortly feed 
back to colleges information about students’ achievements in a form that will 
enable them to compare their performance with that of other ‘like colleges’. 

There are also plans to award some colleges accredited status. This is seen 
as a way of enabling the Council to focus its support more effectively on 
colleges experiencing difficulty; but it also reflects what the FEFC describes 
as ‘the growing maturity of the sector’. Accredited status will be awarded to 
colleges demonstrating: 

. . . comprehensive, rigorous and effective systems of management 
control and quality assurance which have a reputation for providing 
high quality educational provision. 

Accredited colleges will be asked to share annual self-assessments with the 
Council, and inspectors will make regular visits to maintain a programme of 
independent inspection and disseminate good practice. Accredited status 
may be withdrawn where the accreditation panel feels that the integrity of 
the awards is threatened. 

The schools sector 

Like FEFC and the FE sector, the inspection regime established by OFSTED 
provides the framework for quality improvement work in the schools sector. 
" However, there is not the same sense of there being a single, all-powerful 
body controlling schools. Responsibility for the schools sector has been 
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shared in recent years with other agencies with statutory powers (in particular, 
the Teacher Training Agency and the QCA - until recently the SCAA). The 
schools sector has also been subjected to more overt attention from politicians. 
The result has been the establishment of the strong Standards and Effectiveness 
Unit at the DfEE, which is concerned with quality improvement. 

There has been considerable success in improving the effectiveness of 
schools since 1980. This is clear from GCSE results, which improved 
markedly during the 1980s and continue to improve, albeit at a slower rate. 
Recent research (SCAA, 1996) has not shown any decline in the standards of 
the exam, despite media claims to that effect. 

In addition to inspection, improvement has been driven by the introduction 
of the national curriculum in 1988, tighter controls over initial teacher 
training, careful targeting of money for in-service training of teachers, 
and general erosion of the power of LEAs over schools through a series of 
legislative and financial changes. 

Three main strategies are seen to be driving improvement: 

• greater public accountability of individual schools 
® assessment of individual schools against performance indicators 
® increased autonomy of schools from LEAs, giving them ownership. 

By comparison with these strategies, both curriculum change and teacher 
development are seen as too slow, though fundamentally important. 

Although there are recognisable themes in this approach comparable to 
the FEFC inspection regime, there are also significant differences. The main 
difference is one of style. There is a stronger sense of involvement between 
the FEFC and colleges than between OFSTED and schools. Tensions surround 
the inspection process in colleges, but they are perhaps less extreme than 
those experienced in some schools. Two approaches used by OFSTED are 
indicators of the difference of style: ‘naming and shaming’ of failing schools, 
and placing a failing school on ‘special measures’. Both are highly public 
procedures. The latter involves drafting into the school considerable support, 
with root and branch changes to the organisational structure and key personnel. 
Such measures have only recently been adopted by the FEFC. 
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The improvement process in relation to *special measures* 

During 1993-97, 85 secondary schools judged to be failing were placed on 
special measures (OFSTED, 1998). According to OFSTED: 

For schools to improve from such a low ebb, four prerequisites are necessary: 

• accepting the judgement 

• knowing what to do 

• knowing how to go about it 

• securing the support of all parties: the governors, 
teachers, pupils, parents and the LEA. 

Seven of these schools were judged after two to three years to have improved 
to the point of providing a satisfactory standard of education. OFSTED has 
this to say of the reasons for their rapid improvement: 

Four of the seven appointed a new headteacher early on in the process; 
all made significant changes in the management structure. Changes were 
made to improve the quality of teachers’ planning and the curriculum, 
and rigorous systems for monitoring the quality of teaching were 
implemented, but not always sufficiently early to provide the impetus 
for rapid change. 

Other factors noted by OFSTED as important in achieving their success are: 

• support from the LEA (for the six maintained schools) 

• appointment of additional governors with specialist expertise 

• provision of advice and consultancy, particularly in preparing the 
action plans, monitoring the schools’ progress toward their targets, 
and evaluating the quality of teaching. 

In one school ‘significant funds’ for staff development and building works 
was an important factor. 

There have been some changes in policy since the change of Government in 
1997. It is recognised that schools have experienced problems because of the 
number of things for which they were accountable. The present policy is to 
focus on fewer things, such as numeracy and literacy in the primary school. 
The market philosophy is discredited, because ‘markets tolerate under- 
performance’. It has been replaced by the planning system, in which you: 
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• define the quality standards (national curriculum levels 
of attainment and OFSTED criteria) 

• give responsibility to schools to deliver it 

• hold them accountable, measuring performance 
and reporting this publicly. 

A combined strategy of ‘carrot and stick’ is administered through support 
and inspection processes. The ‘stick’ involves strategies such as ‘naming and 
shaming’ and league tables; these work, even if they may sometimes seem 
unfair. It is not acceptable to protect teachers from public criticism if this 
means withholding information from parents that is known to the profession. 
The ‘carrot’ is support involving funding, providing schools with comparative 
data on their performance in relation to other ‘like’ schools. Other strategies 
have been added recently: beacon schools which will receive additional 
funding in return for providing staff development; knighthoods for outstanding 
headteachers; and awards for exceptional teachers. 

The main performance indicators used by OFSTED are examination results. 
In raw form, these provide the quickest feedback on a school’s performance. 
Value added data is more valuable potentially, but can be ‘too slow’ to 
produce and therefore less useful. Raw examination results provide a direct 
comparison between performance in one year and the next. 

However, OFSTED has carried out significant work on the production 
of value added data for individual schools, working closely with school 
effectiveness researchers at the University of London Institute of Education 
(ULIE). A major review of SER work was commissioned (Sammons, Hillman 
and Mortimore, 1994), and this became the basis for the development of an 
analytical procedure to produce value added data on every school. To produce 
this data, examination results are analysed in relation to other contextual 
information about the school to give a profile of the ‘value added’ by the 
school to its pupils’ attainment. The contextual information has been 
selected on the basis of the ULIE research. It comprises: 

• type of school (grant maintained or not) 

• numbers on free school meals 

® numbers registered with special needs 

• percentage of 15-year-old girls 

• HE qualifications of parents 
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• ethnic origins of pupils 

• whether there is a selective school in the area. 

These are fed back to schools, which of course already have the raw data. 

Improving effectiveness is seen as a complex business: ‘It’s no good just 
fixing one aspect, all aspects have to be fixed’ (interview with a key 
respondent). The biggest single factor that affects pupil attainment is agreed 
to be effective teaching (‘four times the impact of any other factor’). Effective 
teaching within the OFSTED framework is analysed in pedagogical terms 
(e.g. pace and challenge, quality of questioning, feedback to pupils, high 
expectations), rather than in terms of subject knowledge. There is interest in 
research knowledge about factors such as: 

• matching task to pupil 

• time on-task 

• success rate 

• total learning time. 

Also of interest is how these need to be varied to ‘get the right balance’ for 
pupils of different abilities (e.g. less able pupils need to experience a higher 
success rate to maintain motivation). 

Internal and external relationships, quality of management, and quality of 
assessment and its use (particularly ‘diagnostic assessment’) are the three 
next most important factors in terms of effect on pupils’ attainment. By 
comparison, ‘quality of the curriculum’ and ‘staff development and 
appraisal’ have the least effect of the nine factors identified by OFSTED as 
‘contributing to standards of achievement in relation to pupils’ capabilities’. 

The relative importance of these factors in improving school effectiveness 
has been determined by OFSTED, using the data from inspections. Further 
research is needed to determine which factors most improve college effectiveness. 

A key component of school management is seen by OFSTED as the ability to 
use the value added information it provides to inform decision making. An 
MIS is seen, therefore, as an essential tool for managers. Leadership is 
acknowledged to be important. The only quality assurance system widely 
used in schools is Investors in People, which is seen to be useful in encouraging 
teachers to participate and talk to each other. Other systems, such as TQM, 
are seen as: 




64 



62 



fed a report 



V0I2 No 12 



. . . dominated by structures and functions in a way that people would 
find bureaucratic (interview with a key respondent) 

Once effective schools have been identified, good practice can be disseminated 
to other schools. Each school may in some sense be unique, but like can be 
compared with like. In a recent National Foundation for Educational 
Research study (1997) funded by a London LEA, schools were categorised 
according to the relationship between their ‘raw scores’ and ‘value added 
scores’ on examination results. Those with high raw scores and low value 
added scores were seen to be in danger of becoming ‘complacent’ and needing 
to be challenged, and those with high value added scores and low raw scores 
were ‘needing to keep morale high’ (interview with a key respondent). 

The overall message from the schools experience, according to one of our 
participants, is that if you clarify quality and tell people what you want, they will 
deliver it: ‘People may complain but they will move towards those goals.’ 

Proposed policy changes to improve 
the effectiveness of further education 

Our interviews with key participants identified some new directions in 
policy. These are outlined in this section. 

The agenda for improving college effectiveness has a new urgency following 
the publication of the Government Green Paper, The Learning Age (DfEE, 
1998a). The positive response to the Kennedy Report (FEFC, 1997b), 
including promises to inject more money into the EE sector, is matched by 
clear demands for improvement in performance: 

Effective internal quality assurance systems, combined with regular 
independent inspections against a consistent framework of standards, are 
critical to raising quality and achievement and reducing drop-out rates. 

The Learning Age also includes a proposal: 

... to build on the best practice which already exists and harmonise 
post-16 inspection arrangements across schools, FE colleges, LEAs’ adult 
education and training providers. We will work with inspectors and 
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providers and consult widely to develop a national framework and 
common procedures and marking systems. We propose to publish 
appropriate performance indicators and targets for individual providers. 

Two other significant changes to the thrust of policy are intended to 
complement the drive for widening participation set out in the Kennedy 
Report and The Learning Age. They are: 

• a drive to ensure that every student achieves concrete 
outcomes from study in further education 

• a drive to ensure that students are better matched 
to programmes of study. 

These were not identified as major issues by college principals in our question- 
naire survey, suggesting that the new agenda has not yet percolated through 
the system. 

There is a perception that, until now, participation has sometimes been 
valued for its own sake. Although the achievement of qualifications has been 
the stated goal, there has been a tension between this and the open-access 
ethos. Colleges have welcomed allcomers and given students the chance 
to study what they want, without overly strong guidance to dissuade the 
over-ambitious. The culture is now shifting toward concentrating on 
student achievement. 

Current policies are highly challenging to the traditional values of further 
education. For example, one of our respondents offered the view that 
college effectiveness can be seen as ‘a production activity’, in which students 
have a right to acquire the qualification for which they enrol. If one adopts 
this view, there should be ‘zero tolerance for failure’. To achieve this, 10% 
of funding should be used to provide more sophisticated initial assessment 
procedures, for instance, computerised induction programmes (already 
widely used in the US), backed up with better ongoing support to enable 
students with a record of failure to cope with the course. 

The same respondent felt that colleges have improved over recent years, 
but they have improved: 

. . . from an unacceptably low base. The average pass rate for students 
completing a course is low - below 40% in a small but significant 
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minority of colleges. The average pass rate expressed as a percentage of 
those enrolling on a course is probably as low as 50 % nationally. 

The emphasis is now on ensuring effective learning for every student. Group 
sizes need to be increased: the average size of taught groups in further 
education is currently 11 students. Students need to be more fully occupied, 
although not necessarily in taught classes; it is currently possible for a full- 
time student to be in full-time work at the same time. 

Another tension for further education is between providing students with 
choice - they are, after all, attending voluntarily - and giving stronger 
‘guidance’ to ensure that students’ programmes of work address both their 
own needs and those of the local community and UK pic. In considering this 
issue, there is a need to differentiate between 16 to 19-year-olds and adults 
engaged in lifelong learning. 

From our interviews with policy makers, the current view appears to be that 
there should be an ‘entitlement curriculum’ for 16 to 19-year-olds, incorporating 
basic and key skills at one of three levels (academic, vocational, or fairly 
narrowly applied and leading to an apprenticeship). Adults could have 
greater freedom of choice, but the curriculum for 16 to 19-year-olds needs to 
be constrained so that the National Targets for Education and Training can 
be met. To bring additional students into further education, we need to 
create a better environment for study, rather than allowing them to opt for 
inappropriate programmes of study. According to one key respondent, there 
is a need for ‘radical solutions’ to overcome some students’ long history of 
failure, for example by using a football stadium as the venue for study 
(which is currently being tried in Bristol). 

Key points 

® The ways in which inspection is designed to improve effectiveness are 
similar in the schools and FE sectors. There appears to be a convergence 
in approach between the FEFC, FEFCW and OFSTED. 

® In the FE sector, the main purposes of inspection are the validation of 
college self-assessments, the continuous improvement of provision and 
raising of standards, comparison of performance between colleges, and 
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the dissemination of good practice. These are mediated through an 
assessment of teaching and learning, students’ achievement, curriculum 
organisation and management, and staffing and specialist resource provision. 
There is a variation in emphasis between England and Wales. These measures 
are designed to produce a self-critical culture leading to improvement. 

• In the schools sector, there have been three main strategies in the drive to 
improvement: greater public accountability of individual schools; assessment 
of schools against quality standards (attainment targets, exam results and 
OFSTED inspection criteria); and increased responsibility for schools’ 
own actions and policies. 

• In both sectors, data related to value added is collected by the FEFC, 
FEFCW and OFSTED and supplied to schools and colleges for analysis to 
devise policies and practices that will lead to improvement. 

• Of the factors identified as affecting pupil attainment by OFSTED, the 
most important is seen to be effective teaching, with four times the effect 
of any other factor. Other factors that are significant in this respect are 
internal and external relationships, quality of management and quality 
of assessment. The quality of the curriculum and staff development have 
relatively less impact on pupil attainment. The emphasis in the FE sector on 
student attainment is moving towards coping with even greater numbers of 
a greater diversity of clients, and providing them with an educational 
experience relevant to their needs and those of society. 

® Current policy for the FE sector emphasises student attainment, matching 
students to appropriate courses, and improving the quality of teaching 
and learning. 
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Chapter 



Frameworks for research 
and development 



The nature and purpose of frameworks 

A framework for improving educational effectiveness sets out the proce- 
dures for action at all levels, from infrastructure, through organisational 
management, to the work of individual teachers and learners. It is grounded 
in a set of assumptions about educational values, for example the aims and 
purposes of education, including the relative importance of the interests of 
students, their parents, local businesses and the state as a whole. Underpinning 
these values are a variety of theories with explanatory and predictive power, 
ranging from theories about how people learn, to theories of social change 
and economic development. The extent to which these underpinning theories 
are explicit or implicit can vary considerably. As a set of procedures, the 
framework is usually fairly clear to policy makers. However, it is usually 
revealed to teachers and lecturers only through the implementation of a 
range of policies that make up the strategies to put the framework into place. 

This study has revealed the existence of several different frameworks across 
the different educational sectors, partially overlapping, some emanating 
from policy makers at national level, others from policy makers (principals) 
at organisational level. The framework, set by Government, that governs the 
work of the schools sector is the most fully developed, since many of its 
elements have been in place for several years. 
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All frameworks are dynamic rather than fixed, since politicians and policy 
makers continually strive to improve them. All frameworks exist alongside 
other procedures, values and theories grounded in the experience and 
assumptions of the individuals and groups working in the educational 
institutions. It is, therefore, inevitable that any framework will be contested; 
to be effective, it needs to include mechanisms for control. Frameworks are, 
of course, more or less coercive depending on the value they place on individual 
freedom as opposed to ‘the common good’. 

To illustrate the point. Table 3 (pages 72-73) sets out the characteristics of 
two alternative models, or paradigms, for research and development. Captioned 
effectiveness paradigm and improvement paradigm, these are broadly in line 
with two recognisably different approaches to improving educational 
effectiveness. Although very much oversimplified, these two sets of characteristics 
contain all the elements that make up the various frameworks described in 
the earlier chapters. The table might best be read as a set of continua rather 
than a set of opposites, since all the frameworks contain some elements from 
both the effectiveness and the improvement paradigms. 

The existing FEFC and FEFCW frameworks 

The FEFC (and broadly similar FEFCW) procedures for funding and inspection 
currently fall rather more into the effectiveness paradigm than the 
improvement paradigm. They emphasise, for example, identifying good 
practice, inspection, target setting, benchmarking and the measurement of 
performance, and the close links between funding and performance. 

Looking down the list, one can identify further shifts along the continua 
toward the effectiveness paradigm as a result of current policy initiatives. 
These include, for example, the introduction of procedures for analysing 
student retention and achievement and comparing performance between 
one college and another as an integral part of inspection; and the likely 
introduction of mechanisms for publicly identifying failing colleges. 

On the other hand, there have been shifts toward the improvement paradigm. 
One example is the move away from a market-led culture to one of partnership 
and collaboration. 
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In the course of this research, we have collected data that supports two 
different interpretations of the way in which colleges and their staff respond 
to the FEFC’s procedures for improving effectiveness. All colleges are 
worried about the perceived low level of funding. This is certainly as true of 
principals as it is of other staff. However, in relation to inspection and 
related procedures, we have repeatedly been given two different accounts: 

• According to the first account, the revised FEFC inspection framework 
recognises that the sector is ‘maturing’, gives more responsibility to colleges, 
and places more emphasis on partnership rather than control. Relationships 
between the FEFC and colleges are perceived to be good. Many principals 
subscribe to this view. 

• According to the second account, the FEFC has used its funding mechanisms 
and inspection procedures to impose a high level of control on colleges. 
While this has produced the appearance of co-operation, in reality it is 
only compliance and hides deep-seated resistance. Staff have different criteria 
for educational quality from those that guide current FEFC policy and 
management practice. Morale of staff is very low, and the energy and will 
to drive the improvement of effectiveness are sadly lacking. Many, if not 
most, non-managerial staff subscribe to this view. 

In planning an agenda for research and development work to improve 
college effectiveness, it is essential to take both of these accounts seriously. 
They illustrate a tension in the sector; both are true, despite the apparent 
conflict. To neglect the second account would be a mistake that might 
undermine the success of the drive to improve college effectiveness. 

There is also a marked difference between the major priorities for action of 
college principals and those of policy makers. In questionnaire returns, principals 
identified the major issues in the drive to improve college effectiveness as: 

® lack of finance and funding 

• poor pay and conditions for staff leading to low morale 

• lack of management systems 

• lack of training for middle managers 

• increased bureaucracy 

® the need for staff to adopt new methods of teaching, including 
the use of IT for college-based and home delivery. 
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Status of teaching staff • Trained deliverer of learning • Trusted professional 

Motivation of teaching staff • Presumed to be driven by mechanisiris • Presumed to be driven by professional values 

of reward and punishment i 0 • Ownership of change 

• Desire to avoid unpleasant consequences • High morale, self-belief 
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Policy-makers on the other hand, see the main issues as: 

• student achievement 

• better matching of students to courses 

• self-assessment 

• effective teaching and curriculum. 

Although there is some overlap in the area of teaching and learning, this 
would suggest that, despite principals’ positive view of the FEFC, they are 
not fully committed to the new directions of policy. 

There is a need, therefore, to adopt different strategies in relation to some of 
the items in Table 3 (pages 72-73), in particular, the model of change, the 
culture of the organisation and motivation of lecturing staff. In all these 
areas, the assumptions of current FEFC policy and practice are characteristic 
of the effectiveness paradigm. The urgent need is to shift the emphasis 
towards the improvement paradigm in these areas, by increasing staff ownership 
and participation, generating professional values and raising morale 
through increasing the sense of self-worth. If this is done without changing 
the emphasis in other areas, the system will begin to combine the benefits of 
the improvement and effectiveness paradigms. This is difficult to achieve, 
but by no means impossible. 

Key points 

• A framework for improving educational effectiveness sets out the procedures 
for action. It is grounded in a set of assumptions about educational values, 
underpinned by theories that have explanatory and predictive power. 

• Several different frameworks exist across the different educational sectors. 
They are dynamic rather than fixed and, because they are contested, they 
contain mechanisms of control. 

• Frameworks can be categorised, in terms of their characteristic assumptions 
and strategies, according to the extent to which they belong to the effectiveness 
or the improvement model (see Table 3). 
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• FEFC procedures for funding and inspection currently fall rather more 
into the effectiveness paradigm than the improvement paradigm, 
although they contain elements of both. Current policy changes show 
some shifts in both directions. 

• Colleges are worried about the perceived low level of funding. Although 
principals are generally positive about the revised FEFC inspection frame- 
work, and staff appear to be co-operating, there is evidence that this may be 
little more than compliance and hides deep-seated resistance. Low morale is 
sapping the energy and will to drive the improvement of effectiveness. 

• This can be addressed by adopting strategies characteristic of the 
improvement paradigm in relation to: the model of change, the culture of 
the organisation and motivation of lecturing staff. The challenge is to do 
this while retaining strategies characteristic of the effectiveness paradigm 
in relation to other areas. 
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Recommendations 



Essential preparatory work 

The points below are fundamental to all subsequent research and development 
activities. They need to be dealt with through a combination of research and 
development work, to draw all staff in the sector into the debate. Although 
the FEFC can already provide working answers to them all, these questions 
need to be given full consideration by the sector as a whole, especially where 
the views of the sector differ from current FEFC policy. 

® There is a need to generate an agreed set of purposes for all colleges to 
which all staff can subscribe. This may necessitate generating agreed sub- 
sets for different ‘families’ of colleges that clearly have different missions. 

• There is a need to agree a definition of effectiveness for all colleges, to 
which all staff can subscribe. This may need to vary according to the college’s 
purposes. ‘Effectiveness’ refers to the relationship between means and 
ends, so an effective college will be one that makes best use of its resources 
to make the greatest possible progress toward its educational goals. SER 
defines this more simply: the more effective school is typically defined as 
one whose students make greater progress over time than comparable 
students in comparable institutions. 
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• Once there is agreement on the purposes of colleges and the definition of 
an effective college, there will be a need to establish the key indicators that 
can be used to judge effectiveness. If possible, this should be a short list. 
On the basis of this scoping study, a selection from the following are proposed 
for consideration: 

Outcomes-based measures 

• student achievement 

• retention of students/low drop-out rates 

• accurate matching of students to courses 
® value for money 

• flexibility of course provision to suit a range of stakeholders 
® employment rates 

Qualitative measures 

® student satisfaction 
® leadership style 
® staff-student relationships 
® community relations 
® ethos/climate/culture 

• quality of teaching and learning. 

® In addition to establishing key indicators of effectiveness, there will be a 
need to establish key indicators of improvement. On the basis of the 
research relating to schools, we suggest that colleges are likely to start to 
change if they work on any or all of the following: 

® the college’s attitude and approach to planning 
® the way the college is run and organised 
® the way the curriculum is organised 
® the ethos or culture of the college 
® aspects of the quality of teaching and learning. 

Future research 

There is an urgent need for both quantitative and qualitative research, 
carried out systematically with representative samples of colleges, to inform 
the development of policy and practice. Up to now, almost no studies have 
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been carried out systematically on a large scale. There has been a tendency 
for research to focus on opportunity samples of colleges, mainly on a voluntary 
basis. Such work is illuminating, but insufficient as the basis for policy. 

It is important to remember that research in this field can never give definitive 
answers. Research in the effectiveness paradigm, particularly the SER 
described in Chapter 6, has particular advantages because it provides a 
quantified basis for action. This research can tell us which characteristics are 
most likely to be a feature of effective schools and their relative importance, 
in statistical terms, compared with one another. Nevertheless, as we said 
earlier, they are a guide to good practice rather than a blueprint for it. Seven 
out of ten schools that do these sorts of things tend to get better results, but 
this is not inevitably the case. 

Qualitative research carried out rigorously can provide more detailed 
insights. In particular, it can provide explanatory theories that are beyond 
the scope of quantitative measures. 

There is also a need for research to be carried out in two different ways: in 
partnership with colleges, and independently of colleges. Participation in the 
process of carrying out research is an effective form of staff development, 
and teacher-researchers provide insights that are not available to ‘outsider’ 
researchers. Often the best way to support effective improvement is to 
involve teachers and lecturers as researchers (Somekh etal., 1992; Somekh, 
1993). However, some research, particularly case study research involving 
careful comparisons between cases, will need to be carried out by independent 
researchers who can design a set of closely comparable studies, as suggested 
byYin (1993; 1994). 

It is important to distinguish between the latter kind of qualitative research, 
which provides robust outcomes comparable in reliability to those produced 
by quantitative research, and the former, which have a different purpose. 

Research within the effectiveness paradigm 

It is important to be clear that the research that needs to be carried out in this 
paradigm within the college sector will build directly on the extensive 
research already carried out within the schools sector. The small number of 
colleges is not sufficient to produce totally new, statistically significant 
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results. While there are 444 colleges, to compare like with like requires 
grouping them in smaller cohorts. However, work carried out with colleges, 
using methods already proven to be robust with the much larger cohort of 
schools, will be of value in establishing trends. 

One effective way forward is to establish longitudinal studies to compare 
‘like now’ with ‘like past’ and ‘like future’. Data is available on the performance 
of all colleges for at least the last three years. This can be compared with 
data collected now and at agreed points in the future. Ideally, this work 
would be on a relatively large scale, involving a representative selection of 
colleges, to provide a systematic longitudinal study of the sector. 

The following questions, which derive from the review of the SER literature 
contained earlier in this report, provide a research agenda. Some of them fall 
into the ‘essential preparatory work’ recommended above; but more detailed 
work will be needed to evaluate initial decisions. 

To enable comparison of like with like 

• Given that different institutions have different mixes of courses, are there 
enough commonalities in terms of outcome measures to make valid and 
useful comparisons? 

• If comparisons do not seem appropriate across all/most institutions, can 
subgroups of institutions be constructed which might more suitably be 
compared because they have the same basic missions? If so, should this be 
done on the basis of purpose and/or location, or on the basis of college 
intakes? 

• Are there any particularly distinctive features of the student constituency 
for FE colleges? An early mapping exercise on some of the major charac- 
teristics of the student populations of different FE colleges might help to 
provide a firmer basis for deciding what kinds of comparisons might 
eventually be suitable. In due course, this might be developed into a 
typology of the different kinds of institutions in the sector. 

To establish key indicators 

• In comparing institutions, which aspects of college performance should 
be taken into account? 
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® Are there any obvious parallels between the factors identified as contributing 
to school effectiveness and those which are likely to affect college effectiveness ? 

® What status should be given to evidence from inspections? The evidence 
from inspections of schools has mirrored that from research in some 
respects. However, there have also been issues where inspectors and 
researchers have adopted rather different procedures. 

® How many outcome measures should be prioritised? Current work is 
limited to a large extent by the relative absence of attention to measures 
other than A-levels, and by the methodological and statistical difficulties 
arising from the client base and assessment regimes of most vocational 
programmes. A good deal more work will be needed to ensure that other 
measures of outcomes have comparable characteristics. 

To enable value added to be taken into consideration 

® What salient aspects of colleges’ intakes could be captured from information 
readily available from existing sources? Would these be sufficient, or would 
further data need to be collected? 

® Could further studies of value added by institutions be undertaken for 
other major qualifications studied in colleges, such as GNVQs? Can any 
comparisons across the different qualifications be usefully made? 

® Are there any systematic differences between the student bodies in different 
colleges that ought to be taken systematically into account? 

To enable improvement to be monitored 

® What is meant by ‘improvement’ in the college context? What criteria might 
usefully be employed and what is available at the current time? 

® Could some colleges that are widely recognised as having improved 
be identified, with a view to exploring the nature of the changes and 
improvements they have undergone? 

® How could suitable estimates of the effectiveness of different departments 
in a college be constructed? 

® Do any particular features of college structures and organisation suggest 
that research should concentrate on variations in institutions rather than 
between them? 
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Research within the improvement paradigm 

This scoping study has identified major gaps in the research so far carried 
out. There is an urgent need for a range of studies to provide a firm knowledge 
base about the sector. 

Case studies of institutions 

There is an urgent need for high-quality case study research to be carried out 
systematically across a representative sample of colleges to: 

• investigate the appropriateness of the outcomes of SER in relation to 
colleges, since the cohort is too small to make it possible to use the SER 
approach alone 

• identify the key processes that drive improvement of college effectiveness, 
and relate these to the characteristics of different kinds of colleges, to 
develop robust explanatory theories. 

A starting point for this research might be the seven characteristics of 
‘goodness’ in American high schools (see Table 2, page 34) derived from the 
research of Lightfoot, a MacArthur Prize Fellow at Harvard University. 
These would form the basis for case study research to determine the character- 
istics of effective colleges. This study, which won the American Educational 
Research Association Award in 1994, provides a model for rigorous case 
study research. Another model for the multi-site case study approach is 
provided by Yin (1993). 

Although colleges might be asked to participate in some aspects of this 
research, it needs to be carefully designed and carried out to a high standard 
to enable useful comparisons between case studies. Such research will 
be sufficiently robust to form the basis for policy development, though 
unuseable for definitive predictions. 

Intervention studies related to quantitative studies 

The proposed longitudinal studies (see above) should incorporate some 
intervention work; this should be researched using qualitative methods, in 
order to enable the monitoring of outcomes resulting from specific interventions. 
There is a particular need for research that tracks students into employment 
after they leave the college and looks at the various strategies that can be 
used to prepare students effectively for work. This information should be 
fed back into colleges to inform curricular provision. 
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other studies 

In the course of carrying out this study, we have been given numerous 
suggestions for specific studies in the improvement paradigm. Others have 
emerged as a result of identifying gaps in the literature. Some ideas are listed 
below, but this should not be seen as a definitive list. 

• Studies into the role of external agencies in improving college effectiveness 
would be helpful in enabling improvement in infrastructure support. 

* Research to enable improvement in management and organisational 
structures might include: characteristics of different styles of leadership 
and their impact on improving college effectiveness; the effect of the ethos 
and culture of the institutions/department on improving college effectiveness; 
and the role of quality assurance systems in improving college effectiveness. 

® Improvement in teaching and learning might be supported by research 
to determine the characteristics of effective teaching and learning, and 
of effective teaching and support for those with disabilities or learning 
difficulties. Evaluations would be useful of: the guidance offered to students, 
and its effectiveness in matching students to courses appropriate to their 
needs, abilities and employment opportunities; the quality of the whole 
student learning experience, including taught time, supported self-study 
and independent working; and the contribution that ICT can make to 
teaching and learning in further education, and identification of strategies 
for using it effectively. 

Future development work 

There is a need for an agency to provide leadership for development work in 
the FE sector. FEDA is already fulfilling this need to an extent and is well 
placed to develop this role. 

Colleges require assistance with target setting and the implementation of 
action plans, which are now requirements of the FEFC. This role would be 
distinct from the monitoring role that is part of the FEFC’s inspection remit. 

In practice, this development work is likely to be most effective if it incorporates 
a research element because of the effect of involvement in research on staff 
development (discussed earlier in this chapter). 




V0I2 No 12 



feda report 



so 



S3 



There is a need for guidelines for carrying out action research into effective 
practice, both by individuals and within departments, to develop methods 
for supporting improvement in practice. Without this work, there is a 
danger that lecturers identified as weak will be left floundering and unable to 
take the necessary action to improve. 

We recommend the establishment of an improvement network involving all 
colleges. Its organisation would need to be given very careful consideration, 
but it should involve all colleges and enable collaboration with a ‘like’ 
partner and dissemination of good practice between more and less effective 
colleges. It has been suggested that an improvement network might: 

• operate on a regional basis 

• require all colleges in the region to establish a partnership 
with a ‘like’ college (judged by the FEFC analysis of the 
individualised student records data) 

• require all departments in a partnership teaching the same 
subjects to allocate some time and resources to working 
together for mutual learning 

• provide a framework of research activities for colleges 
to carry out in their partnerships 

• require all partnerships to group with two other partnerships 
(to make up six colleges, spanning the ‘more effective’, ‘average’ 
and ‘less effective’ according to the FEFC analysis); and to 
develop an annual programme of shared events 

• hold regional conferences for sharing the outcomes 
from the research studies 

• publish succinct, high quality accounts of the research studies and 
their outcomes for circulation to all colleges and for use within 
teacher training and staff development. 

There is also a need for support and consultancy to be made available at the 
request of colleges, to help them improve their effectiveness. 

Finally, there is a need for colleges to be visited regularly to identify and 
record instances of good practice. This must be done with care, since not all 
that passes for good practice may currently be worthy of the description. 
Strategies for disseminating good practice should then be developed, using a 
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range of methods including accounts on the web and paper-based publications. 

A database of accounts of good practice and related research should be 

established and made available to all colleges via the internet. 

Key points 

Essential preparatory work 

• The following work is fundamental to all subsequent research and devel- 
opment activities. It requires a combination of research and development 
work to draw all staff in the sector into the debate: 

• generating an agreed set of purposes for all colleges, 
and sub-sets for different ‘families’ of colleges 

• reaching an agreed definition of effectiveness for all colleges, 
with variations for ‘families’ of colleges 

• establishing the key indicators that can be used to judge effectiveness 

• establishing key indicators of improvement. 

Future research 

• There is an urgent need for both quantitative and qualitative research, 
carried out systematically with representative samples of colleges, to inform 
the development of policy and practice. 

® Research can never give definitive answers. 

• Research in the effectiveness paradigm provides a quantified basis for action. 

• Qualitative research provides more detailed insights and explanatory 
theories that are beyond the scope of quantitative measures. 

• There is also a need for research to be carried out: 

• independently of the colleges, because some research, particularly 
case study research which involves careful comparisons between 
cases, needs to be carried out by independent researchers 

• in partnership with the colleges, because participation in the 
process of carrying out research is an effective form of staff 
development and teacher-researchers provide insights not 
available to external researchers. 
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Research within the effectiveness paradigm 

• This work will need to build directly on the extensive research already 
carried out in the schools sector. Work carried out with colleges, using 
methods already proved to be robust with the much larger group of 
schools, will be of value in establishing trends. 

• Longitudinal studies should be set up to compare ‘like now’ with ‘like 
past’ (using existing data) and ‘like future’. These should involve a large 
sample of colleges to provide a systematic longitudinal study of the sector. 

• There is also a need for studies which: establish comparison of like with like; 
establish key indicators; enable value added to be taken into consideration; 
and enable improvement to be monitored. Some of these are included in 
the essential preparatory work (above) but there will be a need for more 
detailed work to evaluate initial decisions. 

Research within the improvement paradigm 

• This scoping study has identified major gaps in the research so far carried 
out. There is an urgent need for a range of studies to provide a firm 
knowledge base about the sector. 

• High quality case study research needs to be carried out systematically 
across a representative sample of colleges to investigate the appropriateness 
of the outcomes of SER in relation to colleges, and to identify and explain 
the key processes which are driving the improvement of college effectiveness. 

• Intervention studies, linked to the longitudinal studies, are needed to 
monitor outcomes resulting from specific interventions. 

• A range of other studies are in the areas of: improvement in infrastructure 
support; improvement in management and organisational structures; and 
improvement in teaching and learning. 




86 



S3 



fed a report 



V0I2 No 12 



Future development work 

• There is a need for an agency to provide leadership for development work 
in the FE sector; FEDA is well-placed to develop this role. This might 
include assisting colleges with target setting and implementation of action 
plans, which are now requirements of the FEFC. The role would be distinct 
from the monitoring role that is part of the FEFC’s inspection remit. 

® We recommend, in addition: 

® involvement of college staff in research as a form of staff development 
® provision of guidelines for carrying out action research into effective 
practice, to develop methods for supporting improvement in practice 
® establishment of an improvement network of colleges, involving all 
colleges, to enable collaboration with a ‘like’ partner and dissemination 
of good practice between more and less effective colleges 
® support and consultancy at the request of colleges, to help them 
improve their effectiveness 

® identification and recording of good practice; dissemination of good 
practice; and establishment of a database of accounts of good practice 
and related research. 
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Appendices 




Appendix i: Methods of study 



The study was carried out over a 10-week period between mid- April and the 
end of June 1998, by a group of six researchers who, among them, had the 
background and experience in both the FE and schools sectors to access the 
necessary information quickly and easily. Close collaboration between 
members of the FEDA steering group and the university team was maintained 
throughout the period of the study. The university group of researchers was 
supported by an ‘internal advisory group’ of colleagues at the university 
who were carrying out research in related areas. 

Four methods of study were used by the university research team after 
consultation with the FEDA steering group: 

• a literature review in the area of improving the effectiveness 
of educational institutions which encompassed both the 
FE and schools sectors 

• a questionnaire to all FE college principals (including sixth-form colleges) 

• an expert seminar and face-to-face and telephone interviews to elicit the 
opinions of key participants in a position to influence educational policy 

• a focus group with participants at a FEDA forum for middle managers, 
and face-to-face interviews with a range of staff in four colleges. 
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The literature review 



The literature review was divided into two parts. That devoted to the FE 
sector provided the background to work on improving college effectiveness; 
it reviewed a range of research and development activities, much of it carried 
out without any co-ordinated purpose. The work in the schools sector con- 
centrated on reviewing the main themes emerging from the extensive literature 
in the specialist field of SER and highlighting outcomes which might be 
applied in the FE sector. 

The questionnaire 

All college principals in England and Wales received a short, open-ended 
questionnaire informing them about the study and its purposes, and asking 
for their views. This was intended to raise awareness of the study and ensure 
wide-ranging consultation. As expected, the response rate to this was low. 
Initially colleges in Wales and sixth-form colleges were under-represented, 
so a follow-up letter was sent to these two groups, resulting in the return of 
some additional questionnaires. The final response rate was 18%, but was 
representative of the sector as a whole. The poor response rate, which had 
been anticipated, may have been caused by colleges being continually asked 
to supply information to outside bodies. 

The expert seminar and interviews with key respondents 

The original research design included interviews with high-level representatives 
of all major agencies with responsibility for the schools, higher and FE 
sectors, and representatives of employers. The original intention was to 
collect their unique perceptions of progress and future challenges in 
improving the effectiveness of schools and colleges. This was changed to an 
expert seminar to reduce costs. In the event, only representatives from the 
FE sector attended: three principals, an FEFC inspector and a representative 
of the AoC. To cover other agencies we carried out individual face-to-face 
and telephone interviews, lasting between 30 and 60 minutes, with key 
players, as follows: 

• a senior inspector from OFSTED 

• a member of the DfEE Standards and Effectiveness Unit 
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• a senior civil servant with responsibility for further education at DfEE 
® a senior representative of the UCOSDA 
® a senior officer from the FEFCW 

® a representative of the Confederation of British Industry 

Focus group and interviews with a range of college staff 

The original research design included telephone interviews with ‘quality 
managers’ in a sample of colleges. To cut costs, this was to be replaced by focus 
groups arranged to follow FEDA’s Quality Forum meetings. The researchers 
were able to attend only one of these meetings, during which a focus group 
was established, notes were taken and a questionnaire distributed and collected. 
The co-ordinators of the other Quality Forums were asked to form a focus 
group and distribute the questionnaires at their next meeting. Where a focus 
group could not be established, co-ordinators were asked to distribute the 
questionnaires, which would be returned either to them or the university. Only 
three questionnaires were returned in this part of the exercise. To compensate 
for this very poor response, resulting in part from logistical difficulties, two 
of the researchers undertook face-to-face interviews with a range of staff in 
four colleges. 
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Professor Bridget Somekh, University of Huddersfield 

Professor Bridget Somekh, who was formerly the Deputy Director of the 
Scottish Council for Research in Education, is an experienced researcher 
and evaluator. She has published widely in the fields of self-evaluation and 
the management of change, the latter at both individual and organisational 
levels. During the 1980s, while at the Cambridge Institute of Education, she 
worked with Suffolk and Norfolk LEAs on an MSC-funded project to support 
the implementation of innovations in large secondary schools. Later, at the 
University of East Anglia, she led two major projects funded by the National 
Council for Educational Technology. These focused on improving the use of 
IT as a tool for learning in, respectively, schools and teacher training institutions. 
In Scotland she was involved in several Government-sponsored evaluations, 
including the National Record of Achievement in Scotland (1997), the 
Standard Entrance Test for the Scottish Police (1996), the Career Entry 
Profiles Pilot Project for the Teacher Training Agency (1996) and the 
Scottish projects in the Education Department’s Superhighways Initiative 
(1998). Professor Somekh’s present position is Dean of the School of 
Education and Professional Development at the University of Huddersfield. 

Dr Andy Convery Redcar and Cleveland College 

Dr Andy Convery has an extensive research background using qualitative 
methods. His recent research interests focus on practitioner research and 
collaborative enquiry between teachers, and promoting reflective teaching 
practice in PE colleges. At present he is actively engaged in exploring how to 
encourage college-wide practitioner research and how this can contribute to 
a self-assessing culture in colleges. He has a publication in press in which he 
considers how links with higher education can promote institutional devel- 
opment in further education. 
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Jean Delany retired as a senior lecturer in Vocational Education in Marcfi 
1997, having completed 25 years’ service in teacher training aimed at the 
post-compulsory sector. She has continued as a senior research associate. In 
1994, she was invited to attend the training events for registered FEFC 
Inspectors and was subsequently added to the Yorkshire and Humberside 
Regional register, taking part in significant number of inspections related to 
the Hotel and Catering specialist area, during the first four-year cycle. She 
continues to be involved under the revised inspection arrangements. Her 
main area of interest focuses on research into quality of the learning process 
and work-based learning and assessment. Jean Delaney has strong links 
with colleges of further education in a specialist context and, through the 
inspection process, has assisted colleges in their preparation for inspection as 
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in respected journals. He is currently a member of the British Educational 
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Professor John Gray (MA Oxon, EdM Harvard, DPhil Sussex) is Director of 
Research at Homerton College, Cambridge and a Visiting Professor at the 
London University Institute of Education. He was previously Professor of 
Education at Sheffield University. He has served on various committees of 
the Economic and Social Research Council connected with education and 
has been a member of numerous steering groups for Government-funded 
research projects. He has held grants for policy-related research from organ- 
isations including the Audit Commission, the DfEE, various LEAs and the 
Scottish Office. A leading researcher in the fields of effectiveness and 
improvement, he has made influential contributions to the creation of value- 
added approaches to judging performance, institutional evaluation, transitions 
from school to training and further education, and the effect of inspection 
on institutional development. He is currently completing a major ESRC- 
funded study of ‘improving’ schools. He is the author of five books and some 
150 other publications. 
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Early in his career, Stan Gunn had extensive experience of evaluation studies 
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has carried out an evaluation of departmental structures for an EE college. 
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which has become his specialism, with an expertise in quantitative methods. 
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Dr Andrew Henworth, Bishop Burton College 

Dr Andrew Henworth is a lecturer in agriculture at Bishop Burton College. 
His main research interest is into the demands for quality in further edu- 
cation. His masters degree concentrated on evaluating and reviewing further 
education and training. For his PhD, he investigated quality in further edu- 
cation and training, looking at the demands for quality by the stakeholders in 
further education and training. Among the groups he investigated were FEFC, 
BTEC, NCVQ and City and Guilds, as well as students, parents, employers, 
college staff, and ex-students. 

Loraine Powell, University of Huddersfield 

Loraine Powell is Head of the Research Unit on the Management of Change. 
Her main interest lies in Educational Management. Previously, as Project 
Manager for the West Yorkshire Open Learning Federation, she was responsible 
for accreditation procedures in 11 colleges of further education and 12 training 
organisations. Her work as regional educational consultant for a DoE- 
funded project on business/education links gave her responsibility for the 
staff development in eight institutes of higher education in the region. She 
was also responsible for research into cross-institutional evaluations. Her 
current interest is in self-evaluation as a conduit for CPD. 
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Since incorporation, a key demand on colleges 
has been improved efficiency. There is now 
professional and political consensus that 
further education needs to be more effective and 
to focus sharply on the achievements of learners. 
This timely report is based on research for FEDA 
by a team from the University of Huddersfield. 

It contains a robust, critical review of previous 
research, inspection and development activities, 
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effectiveness and improvement in the school 
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for future work. If understanding a problem 
is halfway to resolving it, this report takes 
us some way forward. 
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